[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:27 dist-test-slave-8zvk krb5kdc[19226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:27 dist-test-slave-8zvk krb5kdc[19226](info): set up 2 sockets
Sep 25 02:56:27 dist-test-slave-8zvk krb5kdc[19226](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 02:56:28.782135 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.009s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 02:56:28 dist-test-slave-8zvk krb5kdc[19226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 02:56:28.954133 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:45993
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:45993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:29.393471 19239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:29.393816 19239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:29.393989 19239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:29.401654 19239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:29.401784 19239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:29.401898 19239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:29.401996 19239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:29.402086 19239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:29.412003 19239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:45993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:45993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:29.413489 19239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:29.414901 19239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:29.433179 19245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:29.433707 19247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:29.433776 19244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:29.434369 19239 server_base.cc:1034] running on GCE node
I20260925 02:56:29.435382 19239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:29.437003 19239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:29.438225 19239 hybrid_clock.cc:648] HybridClock initialized: now 1790304989438156 us; error 63 us; skew 500 ppm
I20260925 02:56:29.438643 19239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:29 dist-test-slave-8zvk krb5kdc[19226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304989, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:29.442025 19239 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 02:56:29.443889 19239 webserver.cc:460] Webserver started at http://127.18.196.254:46615/ using document root <none> and password file <none>
I20260925 02:56:29.444670 19239 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:29.444804 19239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:29.445227 19239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:29.448530 19239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "298ecc87b17d4a789297d983ef2c63ff"
format_stamp: "Formatted at 2026-09-25 02:56:29 on dist-test-slave-8zvk"
server_key: "4c7b8410936d70b8eb6d6aa0d13a44d8"
server_key_iv: "91f8276e6b632ec77c38c4bd23bfec48"
server_key_version: "encryptionkey@0"
I20260925 02:56:29.449393 19239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "298ecc87b17d4a789297d983ef2c63ff"
format_stamp: "Formatted at 2026-09-25 02:56:29 on dist-test-slave-8zvk"
server_key: "4c7b8410936d70b8eb6d6aa0d13a44d8"
server_key_iv: "91f8276e6b632ec77c38c4bd23bfec48"
server_key_version: "encryptionkey@0"
I20260925 02:56:29.455717 19239 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260925 02:56:29.459786 19254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:29.461190 19239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:56:29.461431 19239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "298ecc87b17d4a789297d983ef2c63ff"
format_stamp: "Formatted at 2026-09-25 02:56:29 on dist-test-slave-8zvk"
server_key: "4c7b8410936d70b8eb6d6aa0d13a44d8"
server_key_iv: "91f8276e6b632ec77c38c4bd23bfec48"
server_key_version: "encryptionkey@0"
I20260925 02:56:29.461613 19239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:29.476065 19239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:29.480304 19239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:29.480593 19239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:29.503422 19239 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:45993
I20260925 02:56:29.503526 19305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:45993 every 8 connection(s)
I20260925 02:56:29.504988 19239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:29.510094 19306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:29.512508 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 19239
I20260925 02:56:29.512791 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:56:29.514650 19219 external_mini_cluster.cc:1428] Setting key 6651ae3ab9475a92c147408afb106ef2
I20260925 02:56:29.523687 19306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 298ecc87b17d4a789297d983ef2c63ff: Bootstrap starting.
I20260925 02:56:29.527590 19306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 298ecc87b17d4a789297d983ef2c63ff: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:29.528791 19306 log.cc:824] T 00000000000000000000000000000000 P 298ecc87b17d4a789297d983ef2c63ff: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:29.532336 19306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 298ecc87b17d4a789297d983ef2c63ff: No bootstrap required, opened a new log
I20260925 02:56:29.540297 19306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 298ecc87b17d4a789297d983ef2c63ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298ecc87b17d4a789297d983ef2c63ff" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 45993 } }
I20260925 02:56:29.540903 19306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 298ecc87b17d4a789297d983ef2c63ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:29.541090 19306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 298ecc87b17d4a789297d983ef2c63ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 298ecc87b17d4a789297d983ef2c63ff, State: Initialized, Role: FOLLOWER
I20260925 02:56:29.541922 19306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 298ecc87b17d4a789297d983ef2c63ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298ecc87b17d4a789297d983ef2c63ff" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 45993 } }
I20260925 02:56:29.542167 19306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 298ecc87b17d4a789297d983ef2c63ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:29.542409 19306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 298ecc87b17d4a789297d983ef2c63ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:29.542623 19306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 298ecc87b17d4a789297d983ef2c63ff [term 0 FOLLOWER]: Advancing to term 1
Sep 25 02:56:29 dist-test-slave-8zvk krb5kdc[19226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:29.545267 19306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 298ecc87b17d4a789297d983ef2c63ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298ecc87b17d4a789297d983ef2c63ff" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 45993 } }
I20260925 02:56:29.546033 19306 leader_election.cc:302] T 00000000000000000000000000000000 P 298ecc87b17d4a789297d983ef2c63ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 298ecc87b17d4a789297d983ef2c63ff; no voters:
I20260925 02:56:29.546882 19306 leader_election.cc:288] T 00000000000000000000000000000000 P 298ecc87b17d4a789297d983ef2c63ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:29.547045 19311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 298ecc87b17d4a789297d983ef2c63ff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:29.549546 19306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 298ecc87b17d4a789297d983ef2c63ff [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:29.550797 19311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 298ecc87b17d4a789297d983ef2c63ff [term 1 LEADER]: Becoming Leader. State: Replica: 298ecc87b17d4a789297d983ef2c63ff, State: Running, Role: LEADER
I20260925 02:56:29.551710 19311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 298ecc87b17d4a789297d983ef2c63ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298ecc87b17d4a789297d983ef2c63ff" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 45993 } }
I20260925 02:56:29.557504 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:29.518703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55056 (local address 127.18.196.254:45993)
0925 02:56:29.520000 (+ 1297us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:29.520010 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:29.521340 (+ 1330us) server_negotiation.cc:407] Connection header received
0925 02:56:29.531375 (+ 10035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:29.531396 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:29.531752 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:29.532099 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:29.534960 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:29.536402 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:29.538133 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:29.538950 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:29.539165 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:29.545329 (+ 6164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:29.545344 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:29.545358 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:29.545403 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:29.550255 (+ 4852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:29.551469 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:29.551484 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:29.551492 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:29.551746 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:29.552391 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:29.552396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:29.552400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:29.553312 (+ 912us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:29.553689 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:29.554940 (+ 1251us) server_negotiation.cc:299] Negotiation successful
0925 02:56:29.555450 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1090,"thread_start_us":611,"threads_started":1}
I20260925 02:56:29.565423 19313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298ecc87b17d4a789297d983ef2c63ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 298ecc87b17d4a789297d983ef2c63ff. Latest consensus state: current_term: 1 leader_uuid: "298ecc87b17d4a789297d983ef2c63ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298ecc87b17d4a789297d983ef2c63ff" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 45993 } } }
I20260925 02:56:29.566210 19312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298ecc87b17d4a789297d983ef2c63ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "298ecc87b17d4a789297d983ef2c63ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298ecc87b17d4a789297d983ef2c63ff" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 45993 } } }
I20260925 02:56:29.566638 19312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298ecc87b17d4a789297d983ef2c63ff [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:29.565949 19313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298ecc87b17d4a789297d983ef2c63ff [sys.catalog]: This master's current role is: LEADER
W20260925 02:56:29.571976 19325 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 298ecc87b17d4a789297d983ef2c63ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:56:29.572103 19325 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:56:29.572247 19327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:29.574910 19327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:29.588322 19327 catalog_manager.cc:1347] Generated new cluster ID: a3bbc72472e5484f863a7ede425b248c
I20260925 02:56:29.588441 19327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:29.605724 19327 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:29.607268 19327 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:29.613417 19327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 298ecc87b17d4a789297d983ef2c63ff: Generated new TSK 0
I20260925 02:56:29.614284 19327 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
I20260925 02:56:29.672276 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:45993
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:30.046466 19332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:30.046717 19332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:30.046814 19332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:30.054955 19332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:30.055059 19332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:30.055171 19332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 02:56:30.066305 19332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:45993
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:30.067824 19332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:30.069139 19332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:30.082947 19340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:30.083024 19338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:30.082988 19337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:30.084220 19332 server_base.cc:1034] running on GCE node
I20260925 02:56:30.084949 19332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:30.085850 19332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:30.087078 19332 hybrid_clock.cc:648] HybridClock initialized: now 1790304990087032 us; error 54 us; skew 500 ppm
I20260925 02:56:30.087381 19332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:30 dist-test-slave-8zvk krb5kdc[19226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304990, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:30.090543 19332 init.cc:377] Logged in from keytab as kudu/127.18.196.193@KRBTEST.COM (short username kudu)
I20260925 02:56:30.092077 19332 webserver.cc:460] Webserver started at http://127.18.196.193:36897/ using document root <none> and password file <none>
I20260925 02:56:30.092763 19332 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:30.092911 19332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:30.093245 19332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:30.096434 19332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "8ab6198855604d9f8fc545e99db96081"
format_stamp: "Formatted at 2026-09-25 02:56:30 on dist-test-slave-8zvk"
server_key: "e893cc85ee91aebac08b927da65f8b7f"
server_key_iv: "e741df5256f0cd5b79262ae22a6bbab9"
server_key_version: "encryptionkey@0"
I20260925 02:56:30.097404 19332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "8ab6198855604d9f8fc545e99db96081"
format_stamp: "Formatted at 2026-09-25 02:56:30 on dist-test-slave-8zvk"
server_key: "e893cc85ee91aebac08b927da65f8b7f"
server_key_iv: "e741df5256f0cd5b79262ae22a6bbab9"
server_key_version: "encryptionkey@0"
I20260925 02:56:30.102768 19332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 02:56:30.106525 19347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:30.107854 19332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:56:30.108036 19332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "8ab6198855604d9f8fc545e99db96081"
format_stamp: "Formatted at 2026-09-25 02:56:30 on dist-test-slave-8zvk"
server_key: "e893cc85ee91aebac08b927da65f8b7f"
server_key_iv: "e741df5256f0cd5b79262ae22a6bbab9"
server_key_version: "encryptionkey@0"
I20260925 02:56:30.108248 19332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:30.129125 19332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:30.133580 19332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:30.133864 19332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:30.134819 19332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:30.136348 19332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:30.136463 19332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:30.136653 19332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:30.136732 19332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:30.174947 19332 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:37719
I20260925 02:56:30.175035 19459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:37719 every 8 connection(s)
I20260925 02:56:30.176250 19332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 02:56:30.182426 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 19332
I20260925 02:56:30.182674 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 02:56:30.183331 19219 external_mini_cluster.cc:1428] Setting key c2b9e6afc4bb8490eaa1b8578c75a155
Sep 25 02:56:30 dist-test-slave-8zvk krb5kdc[19226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304990, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:30.200420 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:30.179113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:53993 (local address 127.18.196.254:45993)
0925 02:56:30.179775 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:30.179779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:30.180644 (+ 865us) server_negotiation.cc:407] Connection header received
0925 02:56:30.182548 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:30.182554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:30.182648 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:30.182768 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:30.184437 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:30.185579 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:30.188358 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:30.188822 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:30.188936 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:30.193212 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:30.193221 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:30.193229 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:30.193278 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:30.196873 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:30.197645 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:30.197653 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:30.197660 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:30.197827 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:30.198282 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:30.198287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:30.198290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:30.198695 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:30.198897 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:30.199629 (+ 732us) server_negotiation.cc:299] Negotiation successful
0925 02:56:30.199909 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":364,"threads_started":1}
I20260925 02:56:30.201035 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:30.179585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:45993 (local address 127.18.196.193:53993)
0925 02:56:30.180473 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:30.180521 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:30.182273 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:30.183034 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:30.183042 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:30.183602 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:30.184208 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:30.184234 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:30.185728 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:30.185734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:30.188175 (+ 2441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:30.188195 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:30.189101 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:30.189111 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:30.189439 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:30.190193 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:30.190232 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:30.192915 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:30.197025 (+ 4110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:30.197032 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:30.197045 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:30.197500 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:30.197935 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:30.197938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:30.197940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:30.198145 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:30.198828 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:30.198876 (+ 48us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:30.199215 (+ 339us) client_negotiation.cc:769] Sending connection context
0925 02:56:30.199431 (+ 216us) client_negotiation.cc:240] Negotiation successful
0925 02:56:30.199855 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":461,"threads_started":1}
WARNING: no policy specified for kudu/127.18.196.194@KRBTEST.COM; defaulting to no policy
I20260925 02:56:30.202639 19460 heartbeater.cc:347] Connected to a master server at 127.18.196.254:45993
I20260925 02:56:30.203032 19460 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:30.203876 19460 heartbeater.cc:511] Master 127.18.196.254:45993 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.194@KRBTEST.COM" created.
I20260925 02:56:30.206637 19271 ts_manager.cc:194] Registered new tserver with Master: 8ab6198855604d9f8fc545e99db96081 (127.18.196.193:37719)
I20260925 02:56:30.209431 19271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.193@KRBTEST.COM'} at 127.18.196.193:53993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
I20260925 02:56:30.230027 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:45993
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:30.590206 19466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:30.590530 19466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:30.590694 19466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:30.598519 19466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:30.598690 19466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:30.598948 19466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 02:56:30.609717 19466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:45993
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:30.611052 19466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:30.612452 19466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:30.626881 19471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:30.626803 19472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:30.626932 19474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:30.632411 19466 server_base.cc:1034] running on GCE node
I20260925 02:56:30.633013 19466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:30.633919 19466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:30.635105 19466 hybrid_clock.cc:648] HybridClock initialized: now 1790304990635069 us; error 33 us; skew 500 ppm
I20260925 02:56:30.635376 19466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:30 dist-test-slave-8zvk krb5kdc[19226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304990, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:30.638597 19466 init.cc:377] Logged in from keytab as kudu/127.18.196.194@KRBTEST.COM (short username kudu)
I20260925 02:56:30.640141 19466 webserver.cc:460] Webserver started at http://127.18.196.194:39021/ using document root <none> and password file <none>
I20260925 02:56:30.640765 19466 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:30.640861 19466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:30.641109 19466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:30.644083 19466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "c3f2c473aeea4997a255d4d4ade86ccb"
format_stamp: "Formatted at 2026-09-25 02:56:30 on dist-test-slave-8zvk"
server_key: "722b70a9dab0eae31c01c651ac59dd4b"
server_key_iv: "c6933239fad77d44d08957b48ee6a214"
server_key_version: "encryptionkey@0"
I20260925 02:56:30.644814 19466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "c3f2c473aeea4997a255d4d4ade86ccb"
format_stamp: "Formatted at 2026-09-25 02:56:30 on dist-test-slave-8zvk"
server_key: "722b70a9dab0eae31c01c651ac59dd4b"
server_key_iv: "c6933239fad77d44d08957b48ee6a214"
server_key_version: "encryptionkey@0"
I20260925 02:56:30.650094 19466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 02:56:30.653684 19482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:30.654989 19466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:56:30.655151 19466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "c3f2c473aeea4997a255d4d4ade86ccb"
format_stamp: "Formatted at 2026-09-25 02:56:30 on dist-test-slave-8zvk"
server_key: "722b70a9dab0eae31c01c651ac59dd4b"
server_key_iv: "c6933239fad77d44d08957b48ee6a214"
server_key_version: "encryptionkey@0"
I20260925 02:56:30.655390 19466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:30.687351 19466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:30.692054 19466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:30.692291 19466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:30.693190 19466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:30.694514 19466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:30.694586 19466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:30.694669 19466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:30.694721 19466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:30.732358 19466 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:41517
I20260925 02:56:30.732390 19594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:41517 every 8 connection(s)
I20260925 02:56:30.733718 19466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 02:56:30.737771 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 19466
I20260925 02:56:30.737972 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 02:56:30.738550 19219 external_mini_cluster.cc:1428] Setting key 58015a83f09ac0c9362bec7b8673f761
Sep 25 02:56:30 dist-test-slave-8zvk krb5kdc[19226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304990, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.196.195@KRBTEST.COM; defaulting to no policy
I20260925 02:56:30.757802 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:30.736464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:34821 (local address 127.18.196.254:45993)
0925 02:56:30.736946 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:30.736953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:30.740990 (+ 4037us) server_negotiation.cc:407] Connection header received
0925 02:56:30.741056 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:30.741060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:30.741136 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:30.741252 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:30.742871 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:30.744204 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:30.745763 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:30.746605 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:30.746707 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:30.751681 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:30.751690 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:30.751699 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:30.751747 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:30.754653 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:30.755446 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:30.755453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:30.755457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:30.755587 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:30.756053 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:30.756057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:30.756062 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:30.756397 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:30.756551 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:30.757211 (+ 660us) server_negotiation.cc:299] Negotiation successful
0925 02:56:30.757436 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":220,"threads_started":1}
I20260925 02:56:30.758711 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:30.737064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:45993 (local address 127.18.196.194:34821)
0925 02:56:30.737913 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:30.737977 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:30.740629 (+ 2652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:30.741470 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:30.741478 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:30.742036 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:30.742666 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:30.742685 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:30.744369 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:30.744373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:30.745620 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:30.745632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:30.746819 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:30.746828 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:30.747262 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:30.748052 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:30.748083 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:30.751372 (+ 3289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:30.754801 (+ 3429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:30.754810 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:30.754825 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:30.755321 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:30.755711 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:30.755716 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:30.755719 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:30.755932 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:30.756526 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:30.756534 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:30.756863 (+ 329us) client_negotiation.cc:769] Sending connection context
0925 02:56:30.757122 (+ 259us) client_negotiation.cc:240] Negotiation successful
0925 02:56:30.757609 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":335,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.195@KRBTEST.COM" created.
I20260925 02:56:30.760317 19595 heartbeater.cc:347] Connected to a master server at 127.18.196.254:45993
I20260925 02:56:30.760615 19595 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:30.761310 19595 heartbeater.cc:511] Master 127.18.196.254:45993 requested a full tablet report, sending...
I20260925 02:56:30.762814 19271 ts_manager.cc:194] Registered new tserver with Master: c3f2c473aeea4997a255d4d4ade86ccb (127.18.196.194:41517)
I20260925 02:56:30.763702 19271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.194@KRBTEST.COM'} at 127.18.196.194:34821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
I20260925 02:56:30.781819 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:45993
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:31.145334 19601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:31.145731 19601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:31.145879 19601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:31.154199 19601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:31.154320 19601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:31.154439 19601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 02:56:31.165025 19601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:45993
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:31.166402 19601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:31.167855 19601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:31.182955 19606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:31.182914 19607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:31.183130 19609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:31.184150 19601 server_base.cc:1034] running on GCE node
I20260925 02:56:31.184931 19601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:31.185856 19601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:31.187050 19601 hybrid_clock.cc:648] HybridClock initialized: now 1790304991187009 us; error 31 us; skew 500 ppm
I20260925 02:56:31.187327 19601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:31 dist-test-slave-8zvk krb5kdc[19226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304991, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:31.190470 19601 init.cc:377] Logged in from keytab as kudu/127.18.196.195@KRBTEST.COM (short username kudu)
I20260925 02:56:31.192042 19601 webserver.cc:460] Webserver started at http://127.18.196.195:39969/ using document root <none> and password file <none>
I20260925 02:56:31.192703 19601 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:31.192808 19601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:31.193123 19601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:31.196080 19601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "2f4bc01510444358944bbb58266ca6f2"
format_stamp: "Formatted at 2026-09-25 02:56:31 on dist-test-slave-8zvk"
server_key: "63f24640d13f040ca51dc7618450e9ba"
server_key_iv: "7145e1247f1991c9ec6bf275f1f022a1"
server_key_version: "encryptionkey@0"
I20260925 02:56:31.196801 19601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "2f4bc01510444358944bbb58266ca6f2"
format_stamp: "Formatted at 2026-09-25 02:56:31 on dist-test-slave-8zvk"
server_key: "63f24640d13f040ca51dc7618450e9ba"
server_key_iv: "7145e1247f1991c9ec6bf275f1f022a1"
server_key_version: "encryptionkey@0"
I20260925 02:56:31.201821 19601 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 02:56:31.205776 19616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:31.207086 19601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:56:31.207262 19601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "2f4bc01510444358944bbb58266ca6f2"
format_stamp: "Formatted at 2026-09-25 02:56:31 on dist-test-slave-8zvk"
server_key: "63f24640d13f040ca51dc7618450e9ba"
server_key_iv: "7145e1247f1991c9ec6bf275f1f022a1"
server_key_version: "encryptionkey@0"
I20260925 02:56:31.207448 19601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:31.212919 19460 heartbeater.cc:503] Master 127.18.196.254:45993 was elected leader, sending a full tablet report...
I20260925 02:56:31.243790 19601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:31.247823 19601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:31.248117 19601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:31.249075 19601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:31.250649 19601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:31.250779 19601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:31.250941 19601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:31.251003 19601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:31.292696 19601 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:43815
I20260925 02:56:31.292794 19728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:43815 every 8 connection(s)
I20260925 02:56:31.294049 19601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 02:56:31.298455 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 19601
I20260925 02:56:31.298659 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 02:56:31.299283 19219 external_mini_cluster.cc:1428] Setting key 49d86c6afb152e268f37ed4bae7ac390
Sep 25 02:56:31 dist-test-slave-8zvk krb5kdc[19226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304991, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:31.315210 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:31.296719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:57945 (local address 127.18.196.254:45993)
0925 02:56:31.297217 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:31.297221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:31.298897 (+ 1676us) server_negotiation.cc:407] Connection header received
0925 02:56:31.301119 (+ 2222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:31.301123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:31.301189 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:31.301288 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:31.303035 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:31.303808 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:31.305252 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:31.305637 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:31.305723 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:31.309151 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:31.309156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:31.309159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:31.309191 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:31.311989 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:31.312736 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:31.312743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:31.312752 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:31.312919 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:31.313286 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:31.313289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:31.313291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:31.313546 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:31.313716 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:31.314525 (+ 809us) server_negotiation.cc:299] Negotiation successful
0925 02:56:31.314734 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":272,"threads_started":1}
I20260925 02:56:31.316118 19732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:31.297343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:45993 (local address 127.18.196.195:57945)
0925 02:56:31.298710 (+ 1367us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:31.298765 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:31.300874 (+ 2109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:31.301580 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:31.301593 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:31.302262 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:31.302810 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:31.302828 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:31.303947 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:31.303951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:31.305135 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:31.305146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:31.305901 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:31.305906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:31.306172 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:31.306766 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:31.306787 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:31.308979 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:31.312101 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:31.312107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:31.312117 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:31.312601 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:31.313012 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:31.313015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:31.313018 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:31.313187 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:31.313688 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:31.313693 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:31.314103 (+ 410us) client_negotiation.cc:769] Sending connection context
0925 02:56:31.314402 (+ 299us) client_negotiation.cc:240] Negotiation successful
0925 02:56:31.314951 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1059,"thread_start_us":743,"threads_started":1}
I20260925 02:56:31.317368 19729 heartbeater.cc:347] Connected to a master server at 127.18.196.254:45993
I20260925 02:56:31.317651 19729 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:31.318471 19729 heartbeater.cc:511] Master 127.18.196.254:45993 requested a full tablet report, sending...
I20260925 02:56:31.319936 19271 ts_manager.cc:194] Registered new tserver with Master: 2f4bc01510444358944bbb58266ca6f2 (127.18.196.195:43815)
I20260925 02:56:31.320771 19271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.195@KRBTEST.COM'} at 127.18.196.195:57945
I20260925 02:56:31.327541 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:56:31 dist-test-slave-8zvk krb5kdc[19226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304991, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:56:31 dist-test-slave-8zvk krb5kdc[19226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304991, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.196.193@KRBTEST.COM
I20260925 02:56:31.373473 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:31.357046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45684 (local address 127.18.196.193:37719)
0925 02:56:31.357824 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:31.357836 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:31.358226 (+ 390us) server_negotiation.cc:407] Connection header received
0925 02:56:31.358452 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:31.358459 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:31.358680 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:31.358869 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:31.360431 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:31.361485 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:31.362746 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:31.363400 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:31.363543 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:31.367710 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:31.367723 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:31.367737 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:31.367771 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:31.370686 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:31.371370 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:31.371386 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:31.371388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:31.371527 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:31.371914 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:31.371921 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:31.371929 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:31.372252 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:31.372590 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:31.372860 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 02:56:31.373134 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":401,"threads_started":1}
W20260925 02:56:31.374725 19375 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45684
Sep 25 02:56:31 dist-test-slave-8zvk krb5kdc[19226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:56:31 dist-test-slave-8zvk krb5kdc[19226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.193@KRBTEST.COM
I20260925 02:56:31.420452 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:31.406304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34950 (local address 127.18.196.193:37719)
0925 02:56:31.406543 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:31.406561 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:31.407212 (+ 651us) server_negotiation.cc:407] Connection header received
0925 02:56:31.407449 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:31.407453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:31.407567 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:31.407680 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:31.409144 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:31.409900 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:31.411263 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:31.411762 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:31.411902 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:31.416036 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:31.416041 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:31.416044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:31.416077 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:31.418354 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:31.418952 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:31.418956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:31.418958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:31.419078 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:31.419398 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:31.419401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:31.419403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:31.419636 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:31.419807 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:31.420060 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 02:56:31.420218 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 02:56:31.422024 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 19332
I20260925 02:56:31.436030 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:31.424066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34956 (local address 127.18.196.193:37719)
0925 02:56:31.424221 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:31.424225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:31.424508 (+ 283us) server_negotiation.cc:407] Connection header received
0925 02:56:31.424618 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:31.424621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:31.424673 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:31.424750 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:31.425834 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:31.426735 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:31.427991 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:31.428696 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:31.428788 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:31.431098 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:31.431102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:31.431105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:31.431133 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:31.433876 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:31.434457 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:31.434461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:31.434463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:31.434597 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:31.434962 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:31.434966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:31.434968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:31.435202 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:31.435376 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:31.435630 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 02:56:31.435775 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:56:31.436726 19455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:56:31.518103 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 19332
I20260925 02:56:31.534993 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 19466
Sep 25 02:56:31 dist-test-slave-8zvk krb5kdc[19226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.194@KRBTEST.COM
I20260925 02:56:31.549894 19744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:31.535914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.18.196.194:41517)
0925 02:56:31.536495 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:31.536502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:31.536527 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:56:31.536591 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:31.536597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:31.536786 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:31.536917 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:31.538066 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:31.539158 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:31.540277 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:31.540779 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:31.540872 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:31.544737 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:31.544747 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:31.544759 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:31.544791 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:31.547447 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:31.548028 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:31.548034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:31.548037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:31.548151 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:31.548496 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:31.548498 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:31.548500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:31.548809 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:31.549026 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:31.549242 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 02:56:31.549490 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":300,"threads_started":1}
I20260925 02:56:31.551185 19590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:56:31.636617 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 19466
I20260925 02:56:31.659932 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 19601
Sep 25 02:56:31 dist-test-slave-8zvk krb5kdc[19226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.195@KRBTEST.COM
I20260925 02:56:31.676028 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:31.661460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47558 (local address 127.18.196.195:43815)
0925 02:56:31.662163 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:31.662173 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:31.662207 (+ 34us) server_negotiation.cc:407] Connection header received
0925 02:56:31.662281 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:31.662292 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:31.662584 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:31.662738 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:31.663915 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:31.664929 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:31.666141 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:31.666725 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:31.666813 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:31.670633 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:31.670648 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:31.670666 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:31.670717 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:31.673537 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:31.674145 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:31.674151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:31.674153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:31.674261 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:31.674586 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:31.674589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:31.674591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:31.674938 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:31.675160 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:31.675576 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 02:56:31.675752 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":374,"threads_started":1}
I20260925 02:56:31.677178 19724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:56:31.758984 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 19601
I20260925 02:56:31.775342 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 19239
I20260925 02:56:31.776422 19301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:56:31.876012 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 19239
2026-09-25T02:56:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4167 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:31 dist-test-slave-8zvk krb5kdc[19751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:31 dist-test-slave-8zvk krb5kdc[19751](info): set up 2 sockets
Sep 25 02:56:31 dist-test-slave-8zvk krb5kdc[19751](info): commencing operation
krb5kdc: starting...
W20260925 02:56:32.942283 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 02:56:33 dist-test-slave-8zvk krb5kdc[19751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 02:56:33.116317 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:42871
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:42871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:33.423491 19764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:33.423808 19764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:33.424006 19764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:33.430728 19764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:33.430919 19764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:33.431031 19764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:33.431133 19764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:33.431217 19764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:33.440532 19764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:42871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:42871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:33.441699 19764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:33.442983 19764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:33.454171 19772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:33.454329 19769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:33.454427 19770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:33.454980 19764 server_base.cc:1034] running on GCE node
I20260925 02:56:33.455696 19764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:33.456945 19764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:33.458176 19764 hybrid_clock.cc:648] HybridClock initialized: now 1790304993458136 us; error 47 us; skew 500 ppm
I20260925 02:56:33.458519 19764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:33 dist-test-slave-8zvk krb5kdc[19751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304993, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:33.466692 19764 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 02:56:33.468245 19764 webserver.cc:460] Webserver started at http://127.18.196.254:33065/ using document root <none> and password file <none>
I20260925 02:56:33.468891 19764 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:33.469030 19764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:33.469350 19764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:33.472206 19764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "caff8114d31a4e9ea7ab833027d3c504"
format_stamp: "Formatted at 2026-09-25 02:56:33 on dist-test-slave-8zvk"
server_key: "8202a09c25e90963a1909c51653f1650"
server_key_iv: "56e9fcce0950471ae1ffab459debdd69"
server_key_version: "encryptionkey@0"
I20260925 02:56:33.472958 19764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "caff8114d31a4e9ea7ab833027d3c504"
format_stamp: "Formatted at 2026-09-25 02:56:33 on dist-test-slave-8zvk"
server_key: "8202a09c25e90963a1909c51653f1650"
server_key_iv: "56e9fcce0950471ae1ffab459debdd69"
server_key_version: "encryptionkey@0"
I20260925 02:56:33.477679 19764 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 02:56:33.480841 19779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:33.482012 19764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 02:56:33.482182 19764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "caff8114d31a4e9ea7ab833027d3c504"
format_stamp: "Formatted at 2026-09-25 02:56:33 on dist-test-slave-8zvk"
server_key: "8202a09c25e90963a1909c51653f1650"
server_key_iv: "56e9fcce0950471ae1ffab459debdd69"
server_key_version: "encryptionkey@0"
I20260925 02:56:33.482371 19764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:33.508199 19764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:33.512115 19764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:33.512411 19764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:33.529522 19764 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:42871
I20260925 02:56:33.529561 19832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:42871 every 8 connection(s)
I20260925 02:56:33.530735 19764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:33.534113 19833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:33.540751 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 19764
I20260925 02:56:33.540977 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:56:33.541599 19219 external_mini_cluster.cc:1428] Setting key a8288ab60fc323498bbab67b4f153c7a
I20260925 02:56:33.543403 19833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caff8114d31a4e9ea7ab833027d3c504: Bootstrap starting.
I20260925 02:56:33.546471 19833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P caff8114d31a4e9ea7ab833027d3c504: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:33.547482 19833 log.cc:824] T 00000000000000000000000000000000 P caff8114d31a4e9ea7ab833027d3c504: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:33.549932 19833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caff8114d31a4e9ea7ab833027d3c504: No bootstrap required, opened a new log
Sep 25 02:56:33 dist-test-slave-8zvk krb5kdc[19751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:33.557668 19833 raft_consensus.cc:348] T 00000000000000000000000000000000 P caff8114d31a4e9ea7ab833027d3c504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caff8114d31a4e9ea7ab833027d3c504" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 42871 } }
I20260925 02:56:33.557997 19833 raft_consensus.cc:374] T 00000000000000000000000000000000 P caff8114d31a4e9ea7ab833027d3c504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:33.558104 19833 raft_consensus.cc:729] T 00000000000000000000000000000000 P caff8114d31a4e9ea7ab833027d3c504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caff8114d31a4e9ea7ab833027d3c504, State: Initialized, Role: FOLLOWER
I20260925 02:56:33.558702 19833 consensus_queue.cc:260] T 00000000000000000000000000000000 P caff8114d31a4e9ea7ab833027d3c504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caff8114d31a4e9ea7ab833027d3c504" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 42871 } }
I20260925 02:56:33.558980 19833 raft_consensus.cc:388] T 00000000000000000000000000000000 P caff8114d31a4e9ea7ab833027d3c504 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:33.559128 19833 raft_consensus.cc:482] T 00000000000000000000000000000000 P caff8114d31a4e9ea7ab833027d3c504 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:33.559322 19833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P caff8114d31a4e9ea7ab833027d3c504 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:33.562505 19833 raft_consensus.cc:504] T 00000000000000000000000000000000 P caff8114d31a4e9ea7ab833027d3c504 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caff8114d31a4e9ea7ab833027d3c504" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 42871 } }
I20260925 02:56:33.562858 19836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:33.543502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.18.196.254:42871)
0925 02:56:33.544242 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:33.544252 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:33.544289 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:56:33.545915 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:33.545935 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:33.546255 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:33.546587 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:33.547990 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:33.549274 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:33.550499 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:33.551211 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:33.551438 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:33.555390 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:33.555398 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:33.555410 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:33.555446 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:33.558436 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:33.559209 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:33.559217 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:33.559224 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:33.559421 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:33.559793 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:33.559798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:33.559801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:33.560490 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:33.560784 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:33.561187 (+ 403us) server_negotiation.cc:299] Negotiation successful
0925 02:56:33.561604 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":352,"threads_started":1}
I20260925 02:56:33.563093 19833 leader_election.cc:302] T 00000000000000000000000000000000 P caff8114d31a4e9ea7ab833027d3c504 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: caff8114d31a4e9ea7ab833027d3c504; no voters:
I20260925 02:56:33.563730 19833 leader_election.cc:288] T 00000000000000000000000000000000 P caff8114d31a4e9ea7ab833027d3c504 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:33.563836 19838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P caff8114d31a4e9ea7ab833027d3c504 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:33.565131 19838 raft_consensus.cc:686] T 00000000000000000000000000000000 P caff8114d31a4e9ea7ab833027d3c504 [term 1 LEADER]: Becoming Leader. State: Replica: caff8114d31a4e9ea7ab833027d3c504, State: Running, Role: LEADER
I20260925 02:56:33.565660 19838 consensus_queue.cc:237] T 00000000000000000000000000000000 P caff8114d31a4e9ea7ab833027d3c504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caff8114d31a4e9ea7ab833027d3c504" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 42871 } }
I20260925 02:56:33.565819 19833 sys_catalog.cc:564] T 00000000000000000000000000000000 P caff8114d31a4e9ea7ab833027d3c504 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:33.568485 19840 sys_catalog.cc:455] T 00000000000000000000000000000000 P caff8114d31a4e9ea7ab833027d3c504 [sys.catalog]: SysCatalogTable state changed. Reason: New leader caff8114d31a4e9ea7ab833027d3c504. Latest consensus state: current_term: 1 leader_uuid: "caff8114d31a4e9ea7ab833027d3c504" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caff8114d31a4e9ea7ab833027d3c504" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 42871 } } }
I20260925 02:56:33.569232 19839 sys_catalog.cc:455] T 00000000000000000000000000000000 P caff8114d31a4e9ea7ab833027d3c504 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "caff8114d31a4e9ea7ab833027d3c504" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caff8114d31a4e9ea7ab833027d3c504" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 42871 } } }
I20260925 02:56:33.569722 19839 sys_catalog.cc:458] T 00000000000000000000000000000000 P caff8114d31a4e9ea7ab833027d3c504 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:33.569718 19840 sys_catalog.cc:458] T 00000000000000000000000000000000 P caff8114d31a4e9ea7ab833027d3c504 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:33.571152 19847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:33.575500 19847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:33.583395 19847 catalog_manager.cc:1347] Generated new cluster ID: f8f54ad3e4284c09a253a173b4368193
I20260925 02:56:33.583513 19847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:33.606781 19847 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:33.608664 19847 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:33.619108 19847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P caff8114d31a4e9ea7ab833027d3c504: Generated new TSK 0
I20260925 02:56:33.619841 19847 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
I20260925 02:56:33.668630 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:42871
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:33.983143 19859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:33.983436 19859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:33.983593 19859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:33.990777 19859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:33.990942 19859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:33.991110 19859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 02:56:34.000625 19859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:42871
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:34.001878 19859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:34.003230 19859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:34.016000 19865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:34.015962 19867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:34.016659 19864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:34.017360 19859 server_base.cc:1034] running on GCE node
I20260925 02:56:34.017920 19859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:34.018805 19859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:34.020051 19859 hybrid_clock.cc:648] HybridClock initialized: now 1790304994019996 us; error 48 us; skew 500 ppm
I20260925 02:56:34.020359 19859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:34 dist-test-slave-8zvk krb5kdc[19751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304994, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:34.023270 19859 init.cc:377] Logged in from keytab as kudu/127.18.196.193@KRBTEST.COM (short username kudu)
I20260925 02:56:34.024631 19859 webserver.cc:460] Webserver started at http://127.18.196.193:35591/ using document root <none> and password file <none>
I20260925 02:56:34.025260 19859 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:34.025399 19859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:34.025707 19859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:34.028466 19859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "57b61a4018284bc19584bfca8c53a8ec"
format_stamp: "Formatted at 2026-09-25 02:56:34 on dist-test-slave-8zvk"
server_key: "d8eb4b3cdda12a395ba3475b3aa3d186"
server_key_iv: "03d6cf595888acecf7f9f7d5fe3b74b2"
server_key_version: "encryptionkey@0"
I20260925 02:56:34.029218 19859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "57b61a4018284bc19584bfca8c53a8ec"
format_stamp: "Formatted at 2026-09-25 02:56:34 on dist-test-slave-8zvk"
server_key: "d8eb4b3cdda12a395ba3475b3aa3d186"
server_key_iv: "03d6cf595888acecf7f9f7d5fe3b74b2"
server_key_version: "encryptionkey@0"
I20260925 02:56:34.033905 19859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 02:56:34.037201 19874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:34.038396 19859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:56:34.038561 19859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "57b61a4018284bc19584bfca8c53a8ec"
format_stamp: "Formatted at 2026-09-25 02:56:34 on dist-test-slave-8zvk"
server_key: "d8eb4b3cdda12a395ba3475b3aa3d186"
server_key_iv: "03d6cf595888acecf7f9f7d5fe3b74b2"
server_key_version: "encryptionkey@0"
I20260925 02:56:34.038728 19859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:34.063484 19859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:34.067610 19859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:34.067836 19859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:34.068652 19859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:34.070096 19859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:34.070221 19859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:34.070346 19859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:34.070420 19859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:34.101891 19859 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:38671
I20260925 02:56:34.101974 19987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:38671 every 8 connection(s)
I20260925 02:56:34.103334 19859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 02:56:34.104993 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 19859
I20260925 02:56:34.105199 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 02:56:34.106053 19219 external_mini_cluster.cc:1428] Setting key f2c16116f78b001371896d711089fbac
Sep 25 02:56:34 dist-test-slave-8zvk krb5kdc[19751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304994, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.196.194@KRBTEST.COM; defaulting to no policy
I20260925 02:56:34.124780 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:34.105915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:33755 (local address 127.18.196.254:42871)
0925 02:56:34.106388 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:34.106393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:34.109458 (+ 3065us) server_negotiation.cc:407] Connection header received
0925 02:56:34.109509 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:34.109512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:34.109567 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:34.109645 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:34.111681 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:34.112389 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:34.113970 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:34.114668 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:34.114799 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:34.118415 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:34.118423 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:34.118431 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:34.118474 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:34.121953 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:34.122636 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:34.122642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:34.122657 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:34.122791 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:34.123238 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:34.123245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:34.123250 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:34.123590 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:34.123723 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:34.124355 (+ 632us) server_negotiation.cc:299] Negotiation successful
0925 02:56:34.124511 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":265,"threads_started":1}
I20260925 02:56:34.125635 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:34.106558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:42871 (local address 127.18.196.193:33755)
0925 02:56:34.107362 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:34.107416 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:34.109206 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:34.109919 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:34.109932 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:34.110726 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:34.111515 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:34.111541 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:34.112545 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:34.112553 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:34.113828 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:34.113844 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:34.114907 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:34.114914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:34.115106 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:34.115742 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:34.115762 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:34.118198 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:34.122084 (+ 3886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:34.122090 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:34.122100 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:34.122488 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:34.122961 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:34.122965 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:34.122967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:34.123113 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:34.123692 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:34.123697 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:34.124023 (+ 326us) client_negotiation.cc:769] Sending connection context
0925 02:56:34.124243 (+ 220us) client_negotiation.cc:240] Negotiation successful
0925 02:56:34.124567 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":316,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.194@KRBTEST.COM" created.
I20260925 02:56:34.127282 19988 heartbeater.cc:347] Connected to a master server at 127.18.196.254:42871
I20260925 02:56:34.127557 19988 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:34.128192 19988 heartbeater.cc:511] Master 127.18.196.254:42871 requested a full tablet report, sending...
I20260925 02:56:34.130251 19797 ts_manager.cc:194] Registered new tserver with Master: 57b61a4018284bc19584bfca8c53a8ec (127.18.196.193:38671)
I20260925 02:56:34.132464 19797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.193@KRBTEST.COM'} at 127.18.196.193:33755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
I20260925 02:56:34.149495 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:42871
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:34.457976 19994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:34.458225 19994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:34.458393 19994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:34.465531 19994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:34.465629 19994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:34.465806 19994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 02:56:34.475488 19994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:42871
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:34.476684 19994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:34.477977 19994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:34.490911 19999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:34.490945 20002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:34.490985 20000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:34.492224 19994 server_base.cc:1034] running on GCE node
I20260925 02:56:34.493037 19994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:34.494342 19994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:34.495616 19994 hybrid_clock.cc:648] HybridClock initialized: now 1790304994495571 us; error 54 us; skew 500 ppm
I20260925 02:56:34.495967 19994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:34 dist-test-slave-8zvk krb5kdc[19751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304994, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:34.499248 19994 init.cc:377] Logged in from keytab as kudu/127.18.196.194@KRBTEST.COM (short username kudu)
I20260925 02:56:34.500624 19994 webserver.cc:460] Webserver started at http://127.18.196.194:36761/ using document root <none> and password file <none>
I20260925 02:56:34.501253 19994 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:34.501370 19994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:34.501645 19994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:34.504549 19994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "90a3fb33f8024e8a8189c986f5fc86d1"
format_stamp: "Formatted at 2026-09-25 02:56:34 on dist-test-slave-8zvk"
server_key: "6f0f86d5dcaefb9663fa83581f2c8ff4"
server_key_iv: "da292e2571d4204173251d236e493a95"
server_key_version: "encryptionkey@0"
I20260925 02:56:34.505261 19994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "90a3fb33f8024e8a8189c986f5fc86d1"
format_stamp: "Formatted at 2026-09-25 02:56:34 on dist-test-slave-8zvk"
server_key: "6f0f86d5dcaefb9663fa83581f2c8ff4"
server_key_iv: "da292e2571d4204173251d236e493a95"
server_key_version: "encryptionkey@0"
I20260925 02:56:34.510152 19994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:56:34.513450 20009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:34.514698 19994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 02:56:34.514827 19994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "90a3fb33f8024e8a8189c986f5fc86d1"
format_stamp: "Formatted at 2026-09-25 02:56:34 on dist-test-slave-8zvk"
server_key: "6f0f86d5dcaefb9663fa83581f2c8ff4"
server_key_iv: "da292e2571d4204173251d236e493a95"
server_key_version: "encryptionkey@0"
I20260925 02:56:34.515059 19994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:34.530678 19994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:34.534960 19994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:34.535243 19994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:34.536150 19994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:34.537603 19994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:34.537673 19994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:34.537801 19994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:34.537849 19994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:34.571008 19994 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:33859
I20260925 02:56:34.571043 20122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:33859 every 8 connection(s)
I20260925 02:56:34.572235 19994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 02:56:34.573984 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 19994
I20260925 02:56:34.574178 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 02:56:34.574765 19219 external_mini_cluster.cc:1428] Setting key 4525acfff684d1bc49d0a9723506a5de
Sep 25 02:56:34 dist-test-slave-8zvk krb5kdc[19751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304994, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.196.195@KRBTEST.COM; defaulting to no policy
I20260925 02:56:34.593796 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:34.574930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:59609 (local address 127.18.196.254:42871)
0925 02:56:34.575120 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:34.575124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:34.576413 (+ 1289us) server_negotiation.cc:407] Connection header received
0925 02:56:34.578481 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:34.578486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:34.578577 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:34.578687 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:34.580277 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:34.581001 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:34.583378 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:34.583775 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:34.583886 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:34.587924 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:34.587929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:34.587933 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:34.587963 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:34.590595 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:34.591413 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:34.591418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:34.591422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:34.591558 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:34.592007 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:34.592012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:34.592015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:34.592333 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:34.592499 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:34.593185 (+ 686us) server_negotiation.cc:299] Negotiation successful
0925 02:56:34.593425 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.195@KRBTEST.COM" created.
I20260925 02:56:34.595026 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:34.575539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:42871 (local address 127.18.196.194:59609)
0925 02:56:34.576276 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:34.576312 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:34.578222 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:34.578950 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:34.578958 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:34.579521 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:34.580119 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:34.580135 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:34.581156 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:34.581160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:34.583231 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:34.583250 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:34.584017 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:34.584027 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:34.584360 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:34.585105 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:34.585133 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:34.587671 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:34.590784 (+ 3113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:34.590793 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:34.590808 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:34.591274 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:34.591671 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:34.591676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:34.591680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:34.591886 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:34.592465 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:34.592474 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:34.592823 (+ 349us) client_negotiation.cc:769] Sending connection context
0925 02:56:34.593089 (+ 266us) client_negotiation.cc:240] Negotiation successful
0925 02:56:34.593643 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":335,"threads_started":1}
I20260925 02:56:34.596526 20123 heartbeater.cc:347] Connected to a master server at 127.18.196.254:42871
I20260925 02:56:34.596843 20123 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:34.597579 20123 heartbeater.cc:511] Master 127.18.196.254:42871 requested a full tablet report, sending...
I20260925 02:56:34.599148 19797 ts_manager.cc:194] Registered new tserver with Master: 90a3fb33f8024e8a8189c986f5fc86d1 (127.18.196.194:33859)
I20260925 02:56:34.599967 19797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.194@KRBTEST.COM'} at 127.18.196.194:59609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
I20260925 02:56:34.618096 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:42871
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:34.927687 20128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:34.927933 20128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:34.928033 20128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:34.934819 20128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:34.934924 20128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:34.935034 20128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 02:56:34.944288 20128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:42871
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:34.945407 20128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:34.946622 20128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:34.959204 20136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:34.959247 20134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:34.959574 20133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:34.960160 20128 server_base.cc:1034] running on GCE node
I20260925 02:56:34.960662 20128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:34.961486 20128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:34.962677 20128 hybrid_clock.cc:648] HybridClock initialized: now 1790304994962632 us; error 35 us; skew 500 ppm
I20260925 02:56:34.963017 20128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:34 dist-test-slave-8zvk krb5kdc[19751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304994, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:34.965929 20128 init.cc:377] Logged in from keytab as kudu/127.18.196.195@KRBTEST.COM (short username kudu)
I20260925 02:56:34.967233 20128 webserver.cc:460] Webserver started at http://127.18.196.195:45499/ using document root <none> and password file <none>
I20260925 02:56:34.967813 20128 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:34.967905 20128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:34.968153 20128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:34.970659 20128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "8d112bd3ab7447b49bcb1550ad8bac84"
format_stamp: "Formatted at 2026-09-25 02:56:34 on dist-test-slave-8zvk"
server_key: "5c662ecaa64ccbc724c86505977961e3"
server_key_iv: "2ca2ab6815367d1aea416ff6d789b704"
server_key_version: "encryptionkey@0"
I20260925 02:56:34.971532 20128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "8d112bd3ab7447b49bcb1550ad8bac84"
format_stamp: "Formatted at 2026-09-25 02:56:34 on dist-test-slave-8zvk"
server_key: "5c662ecaa64ccbc724c86505977961e3"
server_key_iv: "2ca2ab6815367d1aea416ff6d789b704"
server_key_version: "encryptionkey@0"
I20260925 02:56:34.976087 20128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:56:34.979193 20143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:34.980384 20128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:56:34.980559 20128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "8d112bd3ab7447b49bcb1550ad8bac84"
format_stamp: "Formatted at 2026-09-25 02:56:34 on dist-test-slave-8zvk"
server_key: "5c662ecaa64ccbc724c86505977961e3"
server_key_iv: "2ca2ab6815367d1aea416ff6d789b704"
server_key_version: "encryptionkey@0"
I20260925 02:56:34.980739 20128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:35.005384 20128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:35.009223 20128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:35.009529 20128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:35.010438 20128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:35.011761 20128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:35.011885 20128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:35.012034 20128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:35.012107 20128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:35.045343 20128 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:43825
I20260925 02:56:35.045572 20256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:43825 every 8 connection(s)
I20260925 02:56:35.046533 20128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 02:56:35.054376 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 20128
I20260925 02:56:35.054754 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 02:56:35.055378 19219 external_mini_cluster.cc:1428] Setting key 764c04e08c66e1ed0ee24f2fbd534bc9
Sep 25 02:56:35 dist-test-slave-8zvk krb5kdc[19751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304994, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:35.067248 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:35.049067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:36421 (local address 127.18.196.254:42871)
0925 02:56:35.049273 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:35.049277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:35.050671 (+ 1394us) server_negotiation.cc:407] Connection header received
0925 02:56:35.053049 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:35.053055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:35.053140 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:35.053247 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:35.055177 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.056187 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.058157 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.058641 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.058756 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:35.062009 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:35.062014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:35.062018 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:35.062048 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:35.064670 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:35.065276 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:35.065279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:35.065281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:35.065430 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:35.065841 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:35.065845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:35.065847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:35.066102 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:35.066210 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:35.066783 (+ 573us) server_negotiation.cc:299] Negotiation successful
0925 02:56:35.067020 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 02:56:35.068362 20259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:35.049599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:42871 (local address 127.18.196.195:36421)
0925 02:56:35.050516 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:35.050556 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:35.052775 (+ 2219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:35.053483 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:35.053491 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:35.054012 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:35.054956 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:35.054982 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.056336 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.056341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:35.058007 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:35.058030 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.058882 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.058892 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:35.059107 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:35.059651 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:35.059671 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:35.061826 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:35.064777 (+ 2951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:35.064782 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:35.064792 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:35.065177 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:35.065531 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:35.065534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:35.065536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:35.065706 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:35.066202 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:35.066207 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:35.066484 (+ 277us) client_negotiation.cc:769] Sending connection context
0925 02:56:35.066687 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 02:56:35.067220 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":509,"threads_started":1}
I20260925 02:56:35.069670 20257 heartbeater.cc:347] Connected to a master server at 127.18.196.254:42871
I20260925 02:56:35.069991 20257 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:35.070878 20257 heartbeater.cc:511] Master 127.18.196.254:42871 requested a full tablet report, sending...
I20260925 02:56:35.072238 19797 ts_manager.cc:194] Registered new tserver with Master: 8d112bd3ab7447b49bcb1550ad8bac84 (127.18.196.195:43825)
I20260925 02:56:35.073055 19797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.195@KRBTEST.COM'} at 127.18.196.195:36421
I20260925 02:56:35.082667 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:56:35.107386 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:35.093982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51892 (local address 127.18.196.254:42871)
0925 02:56:35.094166 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:35.094170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:35.094716 (+ 546us) server_negotiation.cc:407] Connection header received
0925 02:56:35.094927 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:35.094932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:35.094994 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:35.095078 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:35.096747 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.097482 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.098729 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.099346 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.099449 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:35.101871 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:35.101876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:35.101879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:35.101910 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:35.104650 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:35.105444 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:35.105448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:35.105450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:35.105566 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:35.105953 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:35.105956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:35.105958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:35.106309 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:35.106476 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:35.106961 (+ 485us) server_negotiation.cc:299] Negotiation successful
0925 02:56:35.107115 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:56:35.117522 19797 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:56:35.125056 19797 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:56:35.135982 19988 heartbeater.cc:503] Master 127.18.196.254:42871 was elected leader, sending a full tablet report...
I20260925 02:56:35.160894 20272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:35.139314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48516 (local address 127.18.196.193:38671)
0925 02:56:35.140378 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:35.140390 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:35.152251 (+ 11861us) server_negotiation.cc:407] Connection header received
0925 02:56:35.152355 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:35.152364 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:35.152604 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:35.153539 (+ 935us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:35.153806 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.154945 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.158269 (+ 3324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.159849 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.160122 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:35.160337 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:35.160414 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 02:56:35.160572 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":832,"thread_start_us":637,"threads_started":1}
I20260925 02:56:35.160930 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:35.146444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:43825 (local address 127.0.0.1:46146)
0925 02:56:35.151988 (+ 5544us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:35.152010 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:35.152152 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:35.153451 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:35.153455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:35.153498 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:35.153830 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:35.153842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.155914 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.155920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:35.158188 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:35.158203 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.159686 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.159692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:35.159878 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:35.159930 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 02:56:35.160089 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 02:56:35.160266 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5361,"mutex_wait_us":62,"thread_start_us":5182,"threads_started":1}
I20260925 02:56:35.161221 20269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:35.137807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:33859 (local address 127.0.0.1:60552)
0925 02:56:35.145982 (+ 8175us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:35.146023 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:35.146187 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:35.151994 (+ 5807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:35.152003 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:35.152031 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:35.152399 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:35.152411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.155232 (+ 2821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.155236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:35.158065 (+ 2829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:35.158082 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.160283 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.160290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:35.160492 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:35.160530 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 02:56:35.160593 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 02:56:35.160718 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7973,"mutex_wait_us":233,"thread_start_us":7853,"threads_started":1}
I20260925 02:56:35.160984 20271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:35.138580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:38671 (local address 127.0.0.1:48516)
0925 02:56:35.150362 (+ 11782us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:35.150385 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:35.150541 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:35.152825 (+ 2284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:35.152832 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:35.152860 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:35.153267 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:35.153279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.155098 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.155105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:35.158105 (+ 3000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:35.158116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.160101 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.160107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:35.160265 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:35.160294 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 02:56:35.160375 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 02:56:35.160550 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11631,"mutex_wait_us":156,"thread_start_us":12046,"threads_started":1}
I20260925 02:56:35.162051 20270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:35.137947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60552 (local address 127.18.196.194:33859)
0925 02:56:35.151294 (+ 13347us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:35.151305 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:35.151339 (+ 34us) server_negotiation.cc:407] Connection header received
0925 02:56:35.151488 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:35.151499 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:35.151771 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:35.151988 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:35.152592 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.155097 (+ 2505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.158269 (+ 3172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.159899 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.161074 (+ 1175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:35.161219 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:35.161299 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 02:56:35.161697 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13147,"thread_start_us":13005,"threads_started":1}
I20260925 02:56:35.162110 20274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:35.146889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46146 (local address 127.18.196.195:43825)
0925 02:56:35.152832 (+ 5943us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:35.152840 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:35.152873 (+ 33us) server_negotiation.cc:407] Connection header received
0925 02:56:35.152983 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:35.152989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:35.153227 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:35.154034 (+ 807us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:35.154374 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.155739 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.158331 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.159490 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.159698 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:35.159885 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:35.161579 (+ 1694us) server_negotiation.cc:299] Negotiation successful
0925 02:56:35.161751 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5737,"thread_start_us":6316,"threads_started":1}
I20260925 02:56:35.164929 19923 tablet_service.cc:1467] Processing CreateTablet for tablet 7b9cdeefc23343d0aff807fa41731b90 (DEFAULT_TABLE table=test-table [id=dd5bc1b39ec3459fb1109c3e7c593231]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:35.165504 20192 tablet_service.cc:1467] Processing CreateTablet for tablet 7b9cdeefc23343d0aff807fa41731b90 (DEFAULT_TABLE table=test-table [id=dd5bc1b39ec3459fb1109c3e7c593231]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:35.165504 20058 tablet_service.cc:1467] Processing CreateTablet for tablet 7b9cdeefc23343d0aff807fa41731b90 (DEFAULT_TABLE table=test-table [id=dd5bc1b39ec3459fb1109c3e7c593231]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:35.166244 19923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b9cdeefc23343d0aff807fa41731b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:35.166776 20192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b9cdeefc23343d0aff807fa41731b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:35.166783 20058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b9cdeefc23343d0aff807fa41731b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:35.177860 20275 tablet_bootstrap.cc:492] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec: Bootstrap starting.
I20260925 02:56:35.181522 20276 tablet_bootstrap.cc:492] T 7b9cdeefc23343d0aff807fa41731b90 P 8d112bd3ab7447b49bcb1550ad8bac84: Bootstrap starting.
I20260925 02:56:35.182309 20275 tablet_bootstrap.cc:654] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:35.182607 20277 tablet_bootstrap.cc:492] T 7b9cdeefc23343d0aff807fa41731b90 P 90a3fb33f8024e8a8189c986f5fc86d1: Bootstrap starting.
I20260925 02:56:35.184093 20275 log.cc:824] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:35.185966 20277 tablet_bootstrap.cc:654] T 7b9cdeefc23343d0aff807fa41731b90 P 90a3fb33f8024e8a8189c986f5fc86d1: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:35.185981 20276 tablet_bootstrap.cc:654] T 7b9cdeefc23343d0aff807fa41731b90 P 8d112bd3ab7447b49bcb1550ad8bac84: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:35.187144 20277 log.cc:824] T 7b9cdeefc23343d0aff807fa41731b90 P 90a3fb33f8024e8a8189c986f5fc86d1: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:35.188550 20276 log.cc:824] T 7b9cdeefc23343d0aff807fa41731b90 P 8d112bd3ab7447b49bcb1550ad8bac84: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:35.190014 20277 tablet_bootstrap.cc:492] T 7b9cdeefc23343d0aff807fa41731b90 P 90a3fb33f8024e8a8189c986f5fc86d1: No bootstrap required, opened a new log
I20260925 02:56:35.190317 20277 ts_tablet_manager.cc:1397] T 7b9cdeefc23343d0aff807fa41731b90 P 90a3fb33f8024e8a8189c986f5fc86d1: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 02:56:35.193351 20275 tablet_bootstrap.cc:492] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec: No bootstrap required, opened a new log
I20260925 02:56:35.193610 20275 ts_tablet_manager.cc:1397] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20260925 02:56:35.198422 20277 raft_consensus.cc:348] T 7b9cdeefc23343d0aff807fa41731b90 P 90a3fb33f8024e8a8189c986f5fc86d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a3fb33f8024e8a8189c986f5fc86d1" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 33859 } } peers { permanent_uuid: "8d112bd3ab7447b49bcb1550ad8bac84" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43825 } } peers { permanent_uuid: "57b61a4018284bc19584bfca8c53a8ec" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38671 } }
I20260925 02:56:35.198916 20277 raft_consensus.cc:374] T 7b9cdeefc23343d0aff807fa41731b90 P 90a3fb33f8024e8a8189c986f5fc86d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:35.199039 20277 raft_consensus.cc:729] T 7b9cdeefc23343d0aff807fa41731b90 P 90a3fb33f8024e8a8189c986f5fc86d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a3fb33f8024e8a8189c986f5fc86d1, State: Initialized, Role: FOLLOWER
I20260925 02:56:35.199611 20277 consensus_queue.cc:260] T 7b9cdeefc23343d0aff807fa41731b90 P 90a3fb33f8024e8a8189c986f5fc86d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a3fb33f8024e8a8189c986f5fc86d1" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 33859 } } peers { permanent_uuid: "8d112bd3ab7447b49bcb1550ad8bac84" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43825 } } peers { permanent_uuid: "57b61a4018284bc19584bfca8c53a8ec" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38671 } }
I20260925 02:56:35.200799 20276 tablet_bootstrap.cc:492] T 7b9cdeefc23343d0aff807fa41731b90 P 8d112bd3ab7447b49bcb1550ad8bac84: No bootstrap required, opened a new log
I20260925 02:56:35.200794 20123 heartbeater.cc:503] Master 127.18.196.254:42871 was elected leader, sending a full tablet report...
I20260925 02:56:35.201191 20276 ts_tablet_manager.cc:1397] T 7b9cdeefc23343d0aff807fa41731b90 P 8d112bd3ab7447b49bcb1550ad8bac84: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.003s
I20260925 02:56:35.201215 20277 ts_tablet_manager.cc:1428] T 7b9cdeefc23343d0aff807fa41731b90 P 90a3fb33f8024e8a8189c986f5fc86d1: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260925 02:56:35.204485 20275 raft_consensus.cc:348] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a3fb33f8024e8a8189c986f5fc86d1" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 33859 } } peers { permanent_uuid: "8d112bd3ab7447b49bcb1550ad8bac84" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43825 } } peers { permanent_uuid: "57b61a4018284bc19584bfca8c53a8ec" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38671 } }
I20260925 02:56:35.204879 20275 raft_consensus.cc:374] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:35.204979 20275 raft_consensus.cc:729] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57b61a4018284bc19584bfca8c53a8ec, State: Initialized, Role: FOLLOWER
I20260925 02:56:35.205546 20275 consensus_queue.cc:260] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a3fb33f8024e8a8189c986f5fc86d1" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 33859 } } peers { permanent_uuid: "8d112bd3ab7447b49bcb1550ad8bac84" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43825 } } peers { permanent_uuid: "57b61a4018284bc19584bfca8c53a8ec" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38671 } }
I20260925 02:56:35.207317 20275 ts_tablet_manager.cc:1428] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260925 02:56:35.211586 20276 raft_consensus.cc:348] T 7b9cdeefc23343d0aff807fa41731b90 P 8d112bd3ab7447b49bcb1550ad8bac84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a3fb33f8024e8a8189c986f5fc86d1" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 33859 } } peers { permanent_uuid: "8d112bd3ab7447b49bcb1550ad8bac84" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43825 } } peers { permanent_uuid: "57b61a4018284bc19584bfca8c53a8ec" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38671 } }
I20260925 02:56:35.211958 20276 raft_consensus.cc:374] T 7b9cdeefc23343d0aff807fa41731b90 P 8d112bd3ab7447b49bcb1550ad8bac84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:35.212081 20276 raft_consensus.cc:729] T 7b9cdeefc23343d0aff807fa41731b90 P 8d112bd3ab7447b49bcb1550ad8bac84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d112bd3ab7447b49bcb1550ad8bac84, State: Initialized, Role: FOLLOWER
I20260925 02:56:35.212649 20276 consensus_queue.cc:260] T 7b9cdeefc23343d0aff807fa41731b90 P 8d112bd3ab7447b49bcb1550ad8bac84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a3fb33f8024e8a8189c986f5fc86d1" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 33859 } } peers { permanent_uuid: "8d112bd3ab7447b49bcb1550ad8bac84" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43825 } } peers { permanent_uuid: "57b61a4018284bc19584bfca8c53a8ec" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38671 } }
I20260925 02:56:35.215636 20257 heartbeater.cc:503] Master 127.18.196.254:42871 was elected leader, sending a full tablet report...
I20260925 02:56:35.216250 20276 ts_tablet_manager.cc:1428] T 7b9cdeefc23343d0aff807fa41731b90 P 8d112bd3ab7447b49bcb1550ad8bac84: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260925 02:56:35.289927 20282 raft_consensus.cc:482] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:56:35.290194 20282 raft_consensus.cc:504] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a3fb33f8024e8a8189c986f5fc86d1" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 33859 } } peers { permanent_uuid: "8d112bd3ab7447b49bcb1550ad8bac84" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43825 } } peers { permanent_uuid: "57b61a4018284bc19584bfca8c53a8ec" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38671 } }
I20260925 02:56:35.292271 20282 leader_election.cc:288] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a3fb33f8024e8a8189c986f5fc86d1 (127.18.196.194:33859), 8d112bd3ab7447b49bcb1550ad8bac84 (127.18.196.195:43825)
W20260925 02:56:35.298307 20258 tablet.cc:2369] T 7b9cdeefc23343d0aff807fa41731b90 P 8d112bd3ab7447b49bcb1550ad8bac84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:56:35.299295 20270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:35.292884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:48509 (local address 127.18.196.194:33859)
0925 02:56:35.293206 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:35.293213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:35.293458 (+ 245us) server_negotiation.cc:407] Connection header received
0925 02:56:35.293718 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:35.293722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:35.293782 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:35.293887 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:35.294661 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.295451 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.297325 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.298235 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.298411 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:35.298500 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:35.298900 (+ 400us) server_negotiation.cc:299] Negotiation successful
0925 02:56:35.299035 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 02:56:35.299319 20284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:35.292674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:33859 (local address 127.18.196.193:48509)
0925 02:56:35.293341 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:35.293357 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:35.293564 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:35.293905 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:35.293910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:35.293934 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:35.294513 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:35.294523 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.295581 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.295589 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:35.297164 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:35.297176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.298485 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.298491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:35.298670 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:35.298704 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 02:56:35.298815 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 02:56:35.298970 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":349,"threads_started":1}
I20260925 02:56:35.300396 20285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:35.293492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:43825 (local address 127.18.196.193:52839)
0925 02:56:35.294540 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:35.294554 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:35.294702 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:35.295055 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:35.295059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:35.295081 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:35.295414 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:35.295423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.296428 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.296435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:35.297930 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:35.297942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.299678 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.299684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:35.299848 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:35.299876 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 02:56:35.299972 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 02:56:35.300089 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"thread_start_us":721,"threads_started":1}
I20260925 02:56:35.300580 20274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:35.293846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:52839 (local address 127.18.196.195:43825)
0925 02:56:35.294228 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:35.294236 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:35.294647 (+ 411us) server_negotiation.cc:407] Connection header received
0925 02:56:35.294862 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:35.294868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:35.294942 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:35.295041 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:35.295574 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.296281 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.298099 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.299448 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.299641 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:35.299715 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:35.300034 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 02:56:35.300205 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260925 02:56:35.300882 20078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9cdeefc23343d0aff807fa41731b90" candidate_uuid: "57b61a4018284bc19584bfca8c53a8ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a3fb33f8024e8a8189c986f5fc86d1" is_pre_election: true
I20260925 02:56:35.301509 20078 raft_consensus.cc:2454] T 7b9cdeefc23343d0aff807fa41731b90 P 90a3fb33f8024e8a8189c986f5fc86d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57b61a4018284bc19584bfca8c53a8ec in term 0.
I20260925 02:56:35.302016 20212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9cdeefc23343d0aff807fa41731b90" candidate_uuid: "57b61a4018284bc19584bfca8c53a8ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d112bd3ab7447b49bcb1550ad8bac84" is_pre_election: true
I20260925 02:56:35.302402 19878 leader_election.cc:302] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57b61a4018284bc19584bfca8c53a8ec, 90a3fb33f8024e8a8189c986f5fc86d1; no voters:
I20260925 02:56:35.302491 20212 raft_consensus.cc:2454] T 7b9cdeefc23343d0aff807fa41731b90 P 8d112bd3ab7447b49bcb1550ad8bac84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57b61a4018284bc19584bfca8c53a8ec in term 0.
I20260925 02:56:35.302783 20282 raft_consensus.cc:2781] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:56:35.302973 20282 raft_consensus.cc:482] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:56:35.303097 20282 raft_consensus.cc:3037] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:35.305581 20282 raft_consensus.cc:504] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a3fb33f8024e8a8189c986f5fc86d1" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 33859 } } peers { permanent_uuid: "8d112bd3ab7447b49bcb1550ad8bac84" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43825 } } peers { permanent_uuid: "57b61a4018284bc19584bfca8c53a8ec" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38671 } }
I20260925 02:56:35.306089 20282 leader_election.cc:288] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec [CANDIDATE]: Term 1 election: Requested vote from peers 90a3fb33f8024e8a8189c986f5fc86d1 (127.18.196.194:33859), 8d112bd3ab7447b49bcb1550ad8bac84 (127.18.196.195:43825)
I20260925 02:56:35.306689 20212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9cdeefc23343d0aff807fa41731b90" candidate_uuid: "57b61a4018284bc19584bfca8c53a8ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d112bd3ab7447b49bcb1550ad8bac84"
I20260925 02:56:35.306744 20078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9cdeefc23343d0aff807fa41731b90" candidate_uuid: "57b61a4018284bc19584bfca8c53a8ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a3fb33f8024e8a8189c986f5fc86d1"
I20260925 02:56:35.306950 20212 raft_consensus.cc:3037] T 7b9cdeefc23343d0aff807fa41731b90 P 8d112bd3ab7447b49bcb1550ad8bac84 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:35.306950 20078 raft_consensus.cc:3037] T 7b9cdeefc23343d0aff807fa41731b90 P 90a3fb33f8024e8a8189c986f5fc86d1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:35.310174 20078 raft_consensus.cc:2454] T 7b9cdeefc23343d0aff807fa41731b90 P 90a3fb33f8024e8a8189c986f5fc86d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57b61a4018284bc19584bfca8c53a8ec in term 1.
I20260925 02:56:35.310174 20212 raft_consensus.cc:2454] T 7b9cdeefc23343d0aff807fa41731b90 P 8d112bd3ab7447b49bcb1550ad8bac84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57b61a4018284bc19584bfca8c53a8ec in term 1.
I20260925 02:56:35.310534 19878 leader_election.cc:302] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57b61a4018284bc19584bfca8c53a8ec, 8d112bd3ab7447b49bcb1550ad8bac84; no voters:
I20260925 02:56:35.310860 20282 raft_consensus.cc:2781] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:35.311285 20282 raft_consensus.cc:686] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec [term 1 LEADER]: Becoming Leader. State: Replica: 57b61a4018284bc19584bfca8c53a8ec, State: Running, Role: LEADER
I20260925 02:56:35.311807 20282 consensus_queue.cc:237] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a3fb33f8024e8a8189c986f5fc86d1" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 33859 } } peers { permanent_uuid: "8d112bd3ab7447b49bcb1550ad8bac84" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43825 } } peers { permanent_uuid: "57b61a4018284bc19584bfca8c53a8ec" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38671 } }
I20260925 02:56:35.316236 19797 catalog_manager.cc:5495] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 57b61a4018284bc19584bfca8c53a8ec (127.18.196.193). New cstate: current_term: 1 leader_uuid: "57b61a4018284bc19584bfca8c53a8ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a3fb33f8024e8a8189c986f5fc86d1" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 33859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d112bd3ab7447b49bcb1550ad8bac84" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57b61a4018284bc19584bfca8c53a8ec" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38671 } health_report { overall_health: HEALTHY } } }
W20260925 02:56:35.324095 20124 tablet.cc:2369] T 7b9cdeefc23343d0aff807fa41731b90 P 90a3fb33f8024e8a8189c986f5fc86d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 02:56:35 dist-test-slave-8zvk krb5kdc[19751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.193@KRBTEST.COM
W20260925 02:56:35.355568 19989 tablet.cc:2369] T 7b9cdeefc23343d0aff807fa41731b90 P 57b61a4018284bc19584bfca8c53a8ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:56:35.356400 20272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:35.340797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48526 (local address 127.18.196.193:38671)
0925 02:56:35.340985 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:35.340989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:35.341714 (+ 725us) server_negotiation.cc:407] Connection header received
0925 02:56:35.341915 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:35.341926 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:35.341988 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:35.342094 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:35.343970 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.344803 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.346031 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.346461 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.346598 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:35.350526 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:35.350539 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:35.350553 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:35.350587 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:35.353417 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:35.354255 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:35.354262 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:35.354264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:35.354453 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:35.354830 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:35.354854 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:35.354857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:35.355215 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:35.355565 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:35.355785 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 02:56:35.356043 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 02:56:35 dist-test-slave-8zvk krb5kdc[19751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304995, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:56:35 dist-test-slave-8zvk krb5kdc[19751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304995, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.196.193@KRBTEST.COM
I20260925 02:56:35.403414 20272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:35.388590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48536 (local address 127.18.196.193:38671)
0925 02:56:35.388842 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:35.388849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:35.389229 (+ 380us) server_negotiation.cc:407] Connection header received
0925 02:56:35.389464 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:35.389470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:35.389547 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:35.389658 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:35.391592 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.392326 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.393615 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.394040 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.394179 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:35.398189 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:35.398195 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:35.398202 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:35.398241 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:35.400681 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:35.401305 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:35.401314 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:35.401316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:35.401526 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:35.401963 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:35.401972 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:35.401974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:35.402216 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:35.402525 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:35.402787 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 02:56:35.403042 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 02:56:35 dist-test-slave-8zvk krb5kdc[19751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:56:35 dist-test-slave-8zvk krb5kdc[19751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.193@KRBTEST.COM
I20260925 02:56:35.448264 20272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:35.433875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48548 (local address 127.18.196.193:38671)
0925 02:56:35.434108 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:35.434115 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:35.434395 (+ 280us) server_negotiation.cc:407] Connection header received
0925 02:56:35.434612 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:35.434616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:35.434691 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:35.434794 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:35.436351 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.437099 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.438255 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.438674 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.438811 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:35.442874 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:35.442883 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:35.442889 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:35.442936 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:35.445544 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:35.446174 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:35.446180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:35.446183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:35.446324 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:35.446748 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:35.446753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:35.446755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:35.447087 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:35.447378 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:35.447660 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 02:56:35.447897 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260925 02:56:35.450088 19903 tablet_service.cc:2908] Rejecting scan request for tablet 7b9cdeefc23343d0aff807fa41731b90: Uninitialized: clean time has not yet been initialized
I20260925 02:56:35.453212 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 19859
I20260925 02:56:35.465042 20272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:35.454358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48564 (local address 127.18.196.193:38671)
0925 02:56:35.454516 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:35.454520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:35.454820 (+ 300us) server_negotiation.cc:407] Connection header received
0925 02:56:35.455013 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:35.455017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:35.455074 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:35.455153 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:35.456235 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.456897 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.457889 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.458225 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.458311 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:35.460237 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:35.460241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:35.460244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:35.460267 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:35.462689 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:35.463403 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:35.463406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:35.463408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:35.463505 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:35.463891 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:35.463894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:35.463896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:35.464141 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:35.464313 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:35.464622 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 02:56:35.464797 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:56:35.465658 19983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:56:35.555276 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 19859
I20260925 02:56:35.573629 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 19994
Sep 25 02:56:35 dist-test-slave-8zvk krb5kdc[19751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.194@KRBTEST.COM
I20260925 02:56:35.589547 20270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:35.574532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.18.196.194:33859)
0925 02:56:35.574693 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:35.574698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:35.574714 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:56:35.574803 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:35.574809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:35.574904 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:35.575025 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:35.576381 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.577134 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.578949 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.579499 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.579686 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:35.584271 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:35.584280 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:35.584292 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:35.584347 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:35.587251 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:35.587865 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:35.587871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:35.587874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:35.587995 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:35.588339 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:35.588343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:35.588345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:35.588703 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:35.588966 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:35.589156 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 02:56:35.589314 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:56:35.590199 20118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:56:35.677873 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 19994
I20260925 02:56:35.694770 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 20128
Sep 25 02:56:35 dist-test-slave-8zvk krb5kdc[19751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.195@KRBTEST.COM
I20260925 02:56:35.708422 20274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:35.695598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46154 (local address 127.18.196.195:43825)
0925 02:56:35.695769 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:35.695774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:35.695792 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:56:35.695958 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:35.695962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:35.696049 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:35.696130 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:35.697352 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.698070 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.699464 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:35.699844 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:35.699934 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:35.703695 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:35.703708 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:35.703723 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:35.703753 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:35.706172 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:35.706778 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:35.706784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:35.706786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:35.706962 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:35.707278 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:35.707281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:35.707283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:35.707597 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:35.707789 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:35.708014 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 02:56:35.708181 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:56:35.709000 20252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:56:35.793251 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 20128
I20260925 02:56:35.810151 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 19764
I20260925 02:56:35.811339 19828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:56:35.918509 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 19764
2026-09-25T02:56:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4042 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:36 dist-test-slave-8zvk krb5kdc[20306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:36 dist-test-slave-8zvk krb5kdc[20306](info): set up 2 sockets
Sep 25 02:56:36 dist-test-slave-8zvk krb5kdc[20306](info): commencing operation
krb5kdc: starting...
W20260925 02:56:36.994269 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 02:56:37 dist-test-slave-8zvk krb5kdc[20306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 02:56:37.191927 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:34431
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:39401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:34431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:37.514247 20319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:37.514606 20319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:37.514767 20319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:37.522058 20319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:37.522197 20319 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 02:56:37.522295 20319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:37.522428 20319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:37.522516 20319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:37.522611 20319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:37.532647 20319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:39401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:34431
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:34431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:37.533949 20319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:37.535279 20319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:37.547039 20324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:37.547264 20327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:37.547133 20325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:37.548161 20319 server_base.cc:1034] running on GCE node
I20260925 02:56:37.548990 20319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:37.550427 20319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:37.551654 20319 hybrid_clock.cc:648] HybridClock initialized: now 1790304997551592 us; error 29 us; skew 500 ppm
I20260925 02:56:37.552008 20319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:37 dist-test-slave-8zvk krb5kdc[20306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304997, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:37.555097 20319 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 02:56:37.556507 20319 webserver.cc:460] Webserver started at http://127.18.196.254:46869/ using document root <none> and password file <none>
I20260925 02:56:37.557152 20319 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:37.557237 20319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:37.557505 20319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:37.560423 20319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "5ef5b7ebf3814ac1ab7bdedcdce87b73"
format_stamp: "Formatted at 2026-09-25 02:56:37 on dist-test-slave-8zvk"
server_key: "23e8683c0607292d42068ca75cc69875"
server_key_iv: "cc541e32d70cb9ad4837bf3cd3593aa5"
server_key_version: "encryptionkey@0"
I20260925 02:56:37.561147 20319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "5ef5b7ebf3814ac1ab7bdedcdce87b73"
format_stamp: "Formatted at 2026-09-25 02:56:37 on dist-test-slave-8zvk"
server_key: "23e8683c0607292d42068ca75cc69875"
server_key_iv: "cc541e32d70cb9ad4837bf3cd3593aa5"
server_key_version: "encryptionkey@0"
I20260925 02:56:37.566179 20319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 02:56:37.569547 20334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:37.570880 20319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:56:37.571043 20319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "5ef5b7ebf3814ac1ab7bdedcdce87b73"
format_stamp: "Formatted at 2026-09-25 02:56:37 on dist-test-slave-8zvk"
server_key: "23e8683c0607292d42068ca75cc69875"
server_key_iv: "cc541e32d70cb9ad4837bf3cd3593aa5"
server_key_version: "encryptionkey@0"
I20260925 02:56:37.571225 20319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:37.585312 20319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:37.589241 20319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:37.589450 20319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:37.611403 20319 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:34431
I20260925 02:56:37.611434 20395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:34431 every 8 connection(s)
I20260925 02:56:37.613263 20319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:37.617456 20396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:37.620246 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 20319
I20260925 02:56:37.620494 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:56:37.621301 19219 external_mini_cluster.cc:1428] Setting key 09c242162c2d0307682ca68d76ecb25f
I20260925 02:56:37.630071 20396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73: Bootstrap starting.
I20260925 02:56:37.633932 20396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:37.635294 20396 log.cc:824] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:56:37 dist-test-slave-8zvk krb5kdc[20306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:37.638288 20396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73: No bootstrap required, opened a new log
I20260925 02:56:37.644544 20399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:37.623464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58058 (local address 127.18.196.254:34431)
0925 02:56:37.624289 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:37.624302 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:37.624344 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:56:37.626044 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:37.626071 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:37.626483 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:37.626823 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:37.628593 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.629882 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.631588 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.632272 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.632400 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:37.637331 (+ 4931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:37.637339 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:37.637354 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:37.637391 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:37.640490 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:37.641212 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:37.641218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:37.641224 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:37.641407 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:37.641814 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:37.641819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:37.641821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:37.642344 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:37.642688 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:37.643150 (+ 462us) server_negotiation.cc:299] Negotiation successful
0925 02:56:37.643554 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":405,"threads_started":1}
I20260925 02:56:37.646265 20396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef5b7ebf3814ac1ab7bdedcdce87b73" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 34431 } }
I20260925 02:56:37.646634 20396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:37.646740 20396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ef5b7ebf3814ac1ab7bdedcdce87b73, State: Initialized, Role: FOLLOWER
I20260925 02:56:37.647356 20396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef5b7ebf3814ac1ab7bdedcdce87b73" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 34431 } }
I20260925 02:56:37.647644 20396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:37.647781 20396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:37.647967 20396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:37.650492 20396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef5b7ebf3814ac1ab7bdedcdce87b73" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 34431 } }
I20260925 02:56:37.651108 20396 leader_election.cc:302] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ef5b7ebf3814ac1ab7bdedcdce87b73; no voters:
I20260925 02:56:37.651827 20396 leader_election.cc:288] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:37.651881 20401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:37.652654 20401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73 [term 1 LEADER]: Becoming Leader. State: Replica: 5ef5b7ebf3814ac1ab7bdedcdce87b73, State: Running, Role: LEADER
I20260925 02:56:37.653301 20401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef5b7ebf3814ac1ab7bdedcdce87b73" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 34431 } }
I20260925 02:56:37.654160 20396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:37.656744 20402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ef5b7ebf3814ac1ab7bdedcdce87b73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef5b7ebf3814ac1ab7bdedcdce87b73" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 34431 } } }
I20260925 02:56:37.656692 20403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ef5b7ebf3814ac1ab7bdedcdce87b73. Latest consensus state: current_term: 1 leader_uuid: "5ef5b7ebf3814ac1ab7bdedcdce87b73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef5b7ebf3814ac1ab7bdedcdce87b73" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 34431 } } }
I20260925 02:56:37.657256 20403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:37.657150 20402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:37.664115 20410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:37.668606 20410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:37.677014 20410 catalog_manager.cc:1347] Generated new cluster ID: 913c9a235a494e76bc491d2f69eb4273
I20260925 02:56:37.677134 20410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:37.685698 20410 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:37.687031 20410 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:37.695104 20410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73: Generated new TSK 0
I20260925 02:56:37.696244 20410 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
I20260925 02:56:37.753974 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:34431
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:39401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:38.083755 20422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:38.083997 20422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:38.084148 20422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:38.091252 20422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:38.091357 20422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:38.091533 20422 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:56:38.091634 20422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 02:56:38.101214 20422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:39401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.196.254:34431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:38.102403 20422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:38.103709 20422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:38.116657 20431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:38.116693 20428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:38.116624 20429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:38.117472 20422 server_base.cc:1034] running on GCE node
I20260925 02:56:38.118040 20422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:38.118985 20422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:38.120219 20422 hybrid_clock.cc:648] HybridClock initialized: now 1790304998120174 us; error 48 us; skew 500 ppm
I20260925 02:56:38.120512 20422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:38 dist-test-slave-8zvk krb5kdc[20306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304998, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:38.123610 20422 init.cc:377] Logged in from keytab as kudu/127.18.196.193@KRBTEST.COM (short username kudu)
I20260925 02:56:38.125077 20422 webserver.cc:460] Webserver started at http://127.18.196.193:34355/ using document root <none> and password file <none>
I20260925 02:56:38.125732 20422 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:38.125861 20422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:38.126180 20422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:38.129022 20422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "052b7e54f5604cde84334d328a43518a"
format_stamp: "Formatted at 2026-09-25 02:56:38 on dist-test-slave-8zvk"
server_key: "4018bafb143404436a9e0951c9033c53"
server_key_iv: "4a0748cdd3ecf98d75b0cdacc025a3cf"
server_key_version: "encryptionkey@0"
I20260925 02:56:38.129824 20422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "052b7e54f5604cde84334d328a43518a"
format_stamp: "Formatted at 2026-09-25 02:56:38 on dist-test-slave-8zvk"
server_key: "4018bafb143404436a9e0951c9033c53"
server_key_iv: "4a0748cdd3ecf98d75b0cdacc025a3cf"
server_key_version: "encryptionkey@0"
I20260925 02:56:38.135004 20422 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 02:56:38.138247 20438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:38.139446 20422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:56:38.139616 20422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "052b7e54f5604cde84334d328a43518a"
format_stamp: "Formatted at 2026-09-25 02:56:38 on dist-test-slave-8zvk"
server_key: "4018bafb143404436a9e0951c9033c53"
server_key_iv: "4a0748cdd3ecf98d75b0cdacc025a3cf"
server_key_version: "encryptionkey@0"
I20260925 02:56:38.139830 20422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:38.169025 20422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:38.173081 20422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:38.173338 20422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:38.176144 20422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:38.176252 20422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:38.176414 20422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:38.176492 20422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:56:38 dist-test-slave-8zvk krb5kdc[20306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304998, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:38.197928 20446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.176470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:43591 (local address 127.18.196.254:34431)
0925 02:56:38.176898 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:38.176918 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:38.178057 (+ 1139us) server_negotiation.cc:407] Connection header received
0925 02:56:38.180138 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:38.180143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:38.180238 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:38.180357 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:38.182607 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.183650 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.186249 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.186787 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.187002 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.191663 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:38.191667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:38.191670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:38.191745 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:38.194420 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:38.195273 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:38.195279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:38.195282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:38.195401 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:38.195982 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:38.195986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:38.195988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:38.196237 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:38.196341 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:38.197344 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0925 02:56:38.197511 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":311,"threads_started":1}
I20260925 02:56:38.199834 20448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.177151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:34431 (local address 127.18.196.193:43591)
0925 02:56:38.177898 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:38.177942 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:38.179871 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:38.180594 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:38.180601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:38.181444 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:38.182432 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:38.182453 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.183809 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.183814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:38.186075 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:38.186095 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.187096 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.187106 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:38.187361 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.188177 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:38.188201 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:38.191400 (+ 3199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:38.194559 (+ 3159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:38.194567 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:38.194581 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:38.195108 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:38.195534 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:38.195542 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:38.195549 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:38.195845 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:38.196366 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:38.196376 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:38.196926 (+ 550us) client_negotiation.cc:769] Sending connection context
0925 02:56:38.197273 (+ 347us) client_negotiation.cc:240] Negotiation successful
0925 02:56:38.197914 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":342,"threads_started":1}
I20260925 02:56:38.224169 20505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.210052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:34431 (local address 127.18.196.193:32975)
0925 02:56:38.210644 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:38.210659 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:38.210821 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:38.211196 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:38.211203 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:38.211907 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:38.212443 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:38.212462 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.213788 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.213794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:38.215403 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:38.215424 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.216545 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.216552 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:38.216693 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.217379 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:38.217395 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:38.218474 (+ 1079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:38.221606 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:38.221611 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:38.221615 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:38.221998 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:38.222435 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:38.222440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:38.222443 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:38.222571 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:38.223145 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:38.223150 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:38.223392 (+ 242us) client_negotiation.cc:769] Sending connection context
0925 02:56:38.223592 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 02:56:38.223849 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":352,"threads_started":1}
I20260925 02:56:38.224352 20446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.209906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:32975 (local address 127.18.196.254:34431)
0925 02:56:38.210045 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:38.210049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:38.210748 (+ 699us) server_negotiation.cc:407] Connection header received
0925 02:56:38.210990 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:38.211002 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:38.211060 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:38.211160 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:38.212602 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.213627 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.215562 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.216125 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.216274 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.218703 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:38.218707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:38.218710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:38.218737 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:38.221467 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:38.222122 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:38.222165 (+ 43us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:38.222168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:38.222316 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:38.222680 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:38.222686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:38.222689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:38.223017 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:38.223178 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:38.223741 (+ 563us) server_negotiation.cc:299] Negotiation successful
0925 02:56:38.223979 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:56:38.232239 20422 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:42561
I20260925 02:56:38.232268 20558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:42561 every 8 connection(s)
I20260925 02:56:38.233412 20422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 02:56:38.235380 20559 heartbeater.cc:347] Connected to a master server at 127.18.196.254:34431
I20260925 02:56:38.235620 20559 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:38.236228 20559 heartbeater.cc:511] Master 127.18.196.254:34431 requested a full tablet report, sending...
I20260925 02:56:38.238063 20351 ts_manager.cc:194] Registered new tserver with Master: 052b7e54f5604cde84334d328a43518a (127.18.196.193:42561)
I20260925 02:56:38.239472 20351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.193@KRBTEST.COM'} at 127.18.196.193:43591
I20260925 02:56:38.242812 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 20422
I20260925 02:56:38.243083 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 02:56:38.243774 19219 external_mini_cluster.cc:1428] Setting key 6a3290d13e1e2e6940b4237be3291679
WARNING: no policy specified for kudu/127.18.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
I20260925 02:56:38.282648 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:34431
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:39401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:38.586673 20563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:38.586922 20563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:38.587087 20563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:38.593663 20563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:38.593779 20563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:38.593952 20563 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:56:38.594079 20563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 02:56:38.602978 20563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:39401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.196.254:34431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:38.604146 20563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:38.605320 20563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:38.617424 20568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:38.619041 20569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:38.619622 20571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:38.620144 20563 server_base.cc:1034] running on GCE node
I20260925 02:56:38.620836 20563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:38.621820 20563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:38.623049 20563 hybrid_clock.cc:648] HybridClock initialized: now 1790304998622994 us; error 30 us; skew 500 ppm
I20260925 02:56:38.623343 20563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:38 dist-test-slave-8zvk krb5kdc[20306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304998, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:38.631700 20563 init.cc:377] Logged in from keytab as kudu/127.18.196.194@KRBTEST.COM (short username kudu)
I20260925 02:56:38.633141 20563 webserver.cc:460] Webserver started at http://127.18.196.194:36453/ using document root <none> and password file <none>
I20260925 02:56:38.633787 20563 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:38.633898 20563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:38.634207 20563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:38.637104 20563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "66f61ee1fa6446f19d3a807a7ca82a42"
format_stamp: "Formatted at 2026-09-25 02:56:38 on dist-test-slave-8zvk"
server_key: "73345f0c0bb26d3ef37e35c4e021778c"
server_key_iv: "01c4466a526eed0d70ca58a1b270c58b"
server_key_version: "encryptionkey@0"
I20260925 02:56:38.637873 20563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "66f61ee1fa6446f19d3a807a7ca82a42"
format_stamp: "Formatted at 2026-09-25 02:56:38 on dist-test-slave-8zvk"
server_key: "73345f0c0bb26d3ef37e35c4e021778c"
server_key_iv: "01c4466a526eed0d70ca58a1b270c58b"
server_key_version: "encryptionkey@0"
I20260925 02:56:38.642915 20563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:56:38.646175 20578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:38.647456 20563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:56:38.647614 20563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "66f61ee1fa6446f19d3a807a7ca82a42"
format_stamp: "Formatted at 2026-09-25 02:56:38 on dist-test-slave-8zvk"
server_key: "73345f0c0bb26d3ef37e35c4e021778c"
server_key_iv: "01c4466a526eed0d70ca58a1b270c58b"
server_key_version: "encryptionkey@0"
I20260925 02:56:38.647814 20563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:38.663543 20563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:38.667243 20563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:38.667500 20563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:38.670099 20563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:38.670199 20563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:38.670327 20563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:38.670409 20563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:56:38 dist-test-slave-8zvk krb5kdc[20306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304998, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:38.694927 20446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.670755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:34323 (local address 127.18.196.254:34431)
0925 02:56:38.670982 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:38.670989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:38.672231 (+ 1242us) server_negotiation.cc:407] Connection header received
0925 02:56:38.674400 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:38.674405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:38.674528 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:38.674647 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:38.676938 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.678238 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.680815 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.681275 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.681459 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.687137 (+ 5678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:38.687144 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:38.687150 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:38.687195 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:38.690682 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:38.691877 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:38.691885 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:38.691890 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:38.692072 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:38.692713 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:38.692721 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:38.692727 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:38.693218 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:38.693406 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:38.694097 (+ 691us) server_negotiation.cc:299] Negotiation successful
0925 02:56:38.694393 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 02:56:38.695494 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.671297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:34431 (local address 127.18.196.194:34323)
0925 02:56:38.672068 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:38.672112 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:38.674063 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:38.674943 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:38.674957 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:38.675736 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:38.676691 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:38.676711 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.678412 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.678421 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:38.680657 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:38.680669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.681594 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.681609 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:38.682026 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.682781 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:38.682825 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:38.686804 (+ 3979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:38.690932 (+ 4128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:38.690944 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:38.690965 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:38.691721 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:38.692205 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:38.692211 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:38.692214 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:38.692547 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:38.693358 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:38.693364 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:38.693693 (+ 329us) client_negotiation.cc:769] Sending connection context
0925 02:56:38.693907 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 02:56:38.694262 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":354,"threads_started":1}
I20260925 02:56:38.718734 20446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.705911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:49235 (local address 127.18.196.254:34431)
0925 02:56:38.706126 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:38.706135 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:38.706474 (+ 339us) server_negotiation.cc:407] Connection header received
0925 02:56:38.706790 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:38.706799 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:38.706948 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:38.707074 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:38.708286 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.708981 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.710218 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.710615 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.710753 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.712600 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:38.712604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:38.712608 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:38.712636 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:38.715780 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:38.716506 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:38.716513 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:38.716517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:38.716706 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:38.717225 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:38.717230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:38.717234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:38.717613 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:38.717783 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:38.718345 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 02:56:38.718515 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:56:38.719384 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.705833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:34431 (local address 127.18.196.194:49235)
0925 02:56:38.706341 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:38.706356 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:38.706602 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:38.707090 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:38.707093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:38.707591 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:38.708146 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:38.708157 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.709116 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.709121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:38.710082 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:38.710093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.710793 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.710799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:38.710984 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.711675 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:38.711691 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:38.712402 (+ 711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:38.715936 (+ 3534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:38.715941 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:38.715945 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:38.716359 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:38.716846 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:38.716866 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:38.716873 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:38.717075 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:38.717760 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:38.717768 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:38.718041 (+ 273us) client_negotiation.cc:769] Sending connection context
0925 02:56:38.718253 (+ 212us) client_negotiation.cc:240] Negotiation successful
0925 02:56:38.718727 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":278,"threads_started":1}
I20260925 02:56:38.727144 20563 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:40003
I20260925 02:56:38.727190 20699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:40003 every 8 connection(s)
I20260925 02:56:38.728276 20563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 02:56:38.730160 20700 heartbeater.cc:347] Connected to a master server at 127.18.196.254:34431
I20260925 02:56:38.730392 20700 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:38.731067 20700 heartbeater.cc:511] Master 127.18.196.254:34431 requested a full tablet report, sending...
I20260925 02:56:38.732306 20351 ts_manager.cc:194] Registered new tserver with Master: 66f61ee1fa6446f19d3a807a7ca82a42 (127.18.196.194:40003)
I20260925 02:56:38.733130 20351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.194@KRBTEST.COM'} at 127.18.196.194:34323
I20260925 02:56:38.737169 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 20563
I20260925 02:56:38.737373 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 02:56:38.737793 19219 external_mini_cluster.cc:1428] Setting key 591e752621984714d9541feeca0b5da6
WARNING: no policy specified for kudu/127.18.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
I20260925 02:56:38.773782 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:34431
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:39401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:39.082460 20704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:39.082716 20704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:39.082909 20704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:39.090123 20704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:39.090236 20704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:39.090431 20704 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:56:39.090559 20704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 02:56:39.100703 20704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:39401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.196.254:34431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:39.101913 20704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:39.103307 20704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:39.115516 20709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:39.115549 20712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:39.115588 20710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:39.116881 20704 server_base.cc:1034] running on GCE node
I20260925 02:56:39.117595 20704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:39.118466 20704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:39.119665 20704 hybrid_clock.cc:648] HybridClock initialized: now 1790304999119594 us; error 44 us; skew 500 ppm
I20260925 02:56:39.119897 20704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:39 dist-test-slave-8zvk krb5kdc[20306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304999, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:39.122761 20704 init.cc:377] Logged in from keytab as kudu/127.18.196.195@KRBTEST.COM (short username kudu)
I20260925 02:56:39.124116 20704 webserver.cc:460] Webserver started at http://127.18.196.195:44605/ using document root <none> and password file <none>
I20260925 02:56:39.124724 20704 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:39.124806 20704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:39.125073 20704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:39.128093 20704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "9311f4c1b9d9425bb385b634ff875406"
format_stamp: "Formatted at 2026-09-25 02:56:39 on dist-test-slave-8zvk"
server_key: "2761cdfee5a9f266f0b5d587a27709fc"
server_key_iv: "5eff065695f6dc7e6b2c438d3771de2b"
server_key_version: "encryptionkey@0"
I20260925 02:56:39.129081 20704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "9311f4c1b9d9425bb385b634ff875406"
format_stamp: "Formatted at 2026-09-25 02:56:39 on dist-test-slave-8zvk"
server_key: "2761cdfee5a9f266f0b5d587a27709fc"
server_key_iv: "5eff065695f6dc7e6b2c438d3771de2b"
server_key_version: "encryptionkey@0"
I20260925 02:56:39.134473 20704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 02:56:39.137732 20719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:39.139077 20704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:56:39.139236 20704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "9311f4c1b9d9425bb385b634ff875406"
format_stamp: "Formatted at 2026-09-25 02:56:39 on dist-test-slave-8zvk"
server_key: "2761cdfee5a9f266f0b5d587a27709fc"
server_key_iv: "5eff065695f6dc7e6b2c438d3771de2b"
server_key_version: "encryptionkey@0"
I20260925 02:56:39.139441 20704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:39.153218 20704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:39.156880 20704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:39.157143 20704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:39.159914 20704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:39.160015 20704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:39.160176 20704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:39.160265 20704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:56:39 dist-test-slave-8zvk krb5kdc[20306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304999, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:39.183413 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:39.160813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:34431 (local address 127.18.196.195:52497)
0925 02:56:39.161682 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:39.161731 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:39.163635 (+ 1904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:39.164415 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:39.164430 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:39.165208 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:39.166058 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:39.166093 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.167853 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.167857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:39.169087 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:39.169099 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.170271 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.170276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:39.170487 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:39.171224 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:39.171257 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:39.175000 (+ 3743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:39.178617 (+ 3617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:39.178628 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:39.178644 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:39.179336 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:39.179864 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:39.179872 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:39.179875 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:39.180087 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:39.180724 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:39.180731 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:39.181200 (+ 469us) client_negotiation.cc:769] Sending connection context
0925 02:56:39.181523 (+ 323us) client_negotiation.cc:240] Negotiation successful
0925 02:56:39.182013 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":379,"threads_started":1}
I20260925 02:56:39.183970 20446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:39.160249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:52497 (local address 127.18.196.254:34431)
0925 02:56:39.160450 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:39.160458 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:39.161865 (+ 1407us) server_negotiation.cc:407] Connection header received
0925 02:56:39.163920 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:39.163926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:39.164019 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:39.164156 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:39.166275 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.167680 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.169228 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.170069 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.170221 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:39.175296 (+ 5075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:39.175305 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:39.175310 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:39.175355 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:39.178473 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:39.179529 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:39.179533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:39.179537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:39.179711 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:39.180209 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:39.180216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:39.180219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:39.180608 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:39.180752 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:39.183123 (+ 2371us) server_negotiation.cc:299] Negotiation successful
0925 02:56:39.183460 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:56:39.202638 20446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:39.189079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:44017 (local address 127.18.196.254:34431)
0925 02:56:39.189258 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:39.189265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:39.189958 (+ 693us) server_negotiation.cc:407] Connection header received
0925 02:56:39.190070 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:39.190078 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:39.190186 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:39.190310 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:39.191222 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.192219 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.194198 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.194786 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.194987 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:39.196591 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:39.196595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:39.196598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:39.196624 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:39.199334 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:39.200296 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:39.200300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:39.200302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:39.200407 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:39.200900 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:39.200908 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:39.200911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:39.201316 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:39.201513 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:39.202075 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 02:56:39.202279 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:56:39.202737 20783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:39.189087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:34431 (local address 127.18.196.195:44017)
0925 02:56:39.189848 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:39.189868 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:39.189976 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:39.190322 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:39.190325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:39.190726 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:39.191077 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:39.191087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.192360 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.192367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:39.194063 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:39.194075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.195021 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.195027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:39.195195 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:39.195796 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:39.195811 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:39.196408 (+ 597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:39.199514 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:39.199522 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:39.199528 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:39.200174 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:39.200542 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:39.200547 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:39.200551 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:39.200753 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:39.201502 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:39.201508 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:39.201771 (+ 263us) client_negotiation.cc:769] Sending connection context
0925 02:56:39.201976 (+ 205us) client_negotiation.cc:240] Negotiation successful
0925 02:56:39.202288 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":399,"threads_started":1}
I20260925 02:56:39.215049 20704 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:38855
I20260925 02:56:39.215085 20838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:38855 every 8 connection(s)
I20260925 02:56:39.216157 20704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 02:56:39.218470 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 20704
I20260925 02:56:39.218681 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 02:56:39.218639 20839 heartbeater.cc:347] Connected to a master server at 127.18.196.254:34431
I20260925 02:56:39.219282 19219 external_mini_cluster.cc:1428] Setting key 0d4be7d4cf83d84cda9fffad885d23d6
I20260925 02:56:39.219360 20839 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:39.220279 20839 heartbeater.cc:511] Master 127.18.196.254:34431 requested a full tablet report, sending...
I20260925 02:56:39.221873 20351 ts_manager.cc:194] Registered new tserver with Master: 9311f4c1b9d9425bb385b634ff875406 (127.18.196.195:38855)
I20260925 02:56:39.223037 20351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.195@KRBTEST.COM'} at 127.18.196.195:52497
I20260925 02:56:39.223870 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:56:39.241811 20559 heartbeater.cc:503] Master 127.18.196.254:34431 was elected leader, sending a full tablet report...
Sep 25 02:56:39 dist-test-slave-8zvk krb5kdc[20306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304999, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:56:39 dist-test-slave-8zvk krb5kdc[20306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304999, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:39.272653 20446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:39.258659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58060 (local address 127.18.196.254:34431)
0925 02:56:39.258912 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:39.258921 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:39.259295 (+ 374us) server_negotiation.cc:407] Connection header received
0925 02:56:39.259541 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:39.259545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:39.259608 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:39.259710 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:39.261070 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.261835 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.263160 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.263617 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.263748 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:39.267510 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:39.267515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:39.267519 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:39.267550 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:39.270021 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:39.270669 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:39.270674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:39.270676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:39.270801 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:39.271233 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:39.271237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:39.271242 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:39.271503 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:39.271750 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:39.272065 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 02:56:39.272299 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:56:39.276793 20350 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:56:39.283432 20350 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:56:39.321134 20856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:39.295990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32900 (local address 127.18.196.193:42561)
0925 02:56:39.312763 (+ 16773us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:39.312775 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:39.314011 (+ 1236us) server_negotiation.cc:407] Connection header received
0925 02:56:39.314232 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:39.314244 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:39.314550 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:39.314745 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:39.315576 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.316916 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.319111 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.319980 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.320165 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:39.320333 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:39.320693 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 02:56:39.320852 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16570,"thread_start_us":16368,"threads_started":1}
I20260925 02:56:39.321200 20854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:39.295069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:42561 (local address 127.0.0.1:32900)
0925 02:56:39.313826 (+ 18757us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:39.313847 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:39.314041 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:39.314942 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:39.314950 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:39.314980 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:39.315358 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:39.315371 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.317076 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.317082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:39.318968 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:39.318981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.320195 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.320201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:39.320474 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:39.320518 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 02:56:39.320638 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 02:56:39.320826 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18587,"mutex_wait_us":56,"thread_start_us":19572,"threads_started":1}
I20260925 02:56:39.323444 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:39.296462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40864 (local address 127.18.196.194:40003)
0925 02:56:39.313250 (+ 16788us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:39.313261 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:39.314998 (+ 1737us) server_negotiation.cc:407] Connection header received
0925 02:56:39.315292 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:39.315301 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:39.315536 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:39.315705 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:39.316541 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.317620 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.320717 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.322097 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.322256 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:39.322402 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:39.322978 (+ 576us) server_negotiation.cc:299] Negotiation successful
0925 02:56:39.323164 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16584,"thread_start_us":16439,"threads_started":1}
I20260925 02:56:39.323453 20853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:39.295349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:40003 (local address 127.0.0.1:40864)
0925 02:56:39.314893 (+ 19544us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:39.314917 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:39.315105 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:39.315869 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:39.315873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:39.315902 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:39.316303 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:39.316318 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.317767 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.317772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:39.320548 (+ 2776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:39.320557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.322534 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.322540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:39.322719 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:39.322753 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 02:56:39.322904 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 02:56:39.323049 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19415,"mutex_wait_us":244,"thread_start_us":18114,"threads_started":1}
I20260925 02:56:39.323944 20852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:39.294714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:38855 (local address 127.0.0.1:47608)
0925 02:56:39.311103 (+ 16389us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:39.311140 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:39.311299 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:39.315267 (+ 3968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:39.315272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:39.315297 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:39.315701 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:39.315714 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.317968 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.317973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:39.320538 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:39.320553 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.323152 (+ 2599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.323160 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:39.323315 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:39.323342 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 02:56:39.323470 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 02:56:39.323592 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16100,"mutex_wait_us":152,"thread_start_us":16821,"threads_started":1}
I20260925 02:56:39.324434 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:39.295672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47608 (local address 127.18.196.195:38855)
0925 02:56:39.314708 (+ 19036us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:39.314716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:39.314749 (+ 33us) server_negotiation.cc:407] Connection header received
0925 02:56:39.314861 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:39.314869 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:39.315097 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:39.315263 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:39.316384 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.317821 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.320721 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.322626 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.322807 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:39.322997 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:39.323974 (+ 977us) server_negotiation.cc:299] Negotiation successful
0925 02:56:39.324124 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18857,"thread_start_us":18648,"threads_started":1}
I20260925 02:56:39.325661 20489 tablet_service.cc:1467] Processing CreateTablet for tablet bd731cd6c08949a092d51b9232397c41 (DEFAULT_TABLE table=test-table [id=fa3c2423f4c042d1b50f2f94a2679936]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:39.326807 20489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd731cd6c08949a092d51b9232397c41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:39.326800 20629 tablet_service.cc:1467] Processing CreateTablet for tablet bd731cd6c08949a092d51b9232397c41 (DEFAULT_TABLE table=test-table [id=fa3c2423f4c042d1b50f2f94a2679936]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:39.328028 20629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd731cd6c08949a092d51b9232397c41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:39.327593 20769 tablet_service.cc:1467] Processing CreateTablet for tablet bd731cd6c08949a092d51b9232397c41 (DEFAULT_TABLE table=test-table [id=fa3c2423f4c042d1b50f2f94a2679936]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:39.328765 20769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd731cd6c08949a092d51b9232397c41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:39.337942 20858 tablet_bootstrap.cc:492] T bd731cd6c08949a092d51b9232397c41 P 052b7e54f5604cde84334d328a43518a: Bootstrap starting.
I20260925 02:56:39.340745 20858 tablet_bootstrap.cc:654] T bd731cd6c08949a092d51b9232397c41 P 052b7e54f5604cde84334d328a43518a: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:39.341670 20859 tablet_bootstrap.cc:492] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42: Bootstrap starting.
I20260925 02:56:39.342780 20858 log.cc:824] T bd731cd6c08949a092d51b9232397c41 P 052b7e54f5604cde84334d328a43518a: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:39.344584 20860 tablet_bootstrap.cc:492] T bd731cd6c08949a092d51b9232397c41 P 9311f4c1b9d9425bb385b634ff875406: Bootstrap starting.
I20260925 02:56:39.345746 20859 tablet_bootstrap.cc:654] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:39.346781 20858 tablet_bootstrap.cc:492] T bd731cd6c08949a092d51b9232397c41 P 052b7e54f5604cde84334d328a43518a: No bootstrap required, opened a new log
I20260925 02:56:39.347106 20858 ts_tablet_manager.cc:1397] T bd731cd6c08949a092d51b9232397c41 P 052b7e54f5604cde84334d328a43518a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260925 02:56:39.347111 20859 log.cc:824] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:39.349632 20859 tablet_bootstrap.cc:492] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42: No bootstrap required, opened a new log
I20260925 02:56:39.349879 20859 ts_tablet_manager.cc:1397] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 02:56:39.351219 20860 tablet_bootstrap.cc:654] T bd731cd6c08949a092d51b9232397c41 P 9311f4c1b9d9425bb385b634ff875406: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:39.352435 20860 log.cc:824] T bd731cd6c08949a092d51b9232397c41 P 9311f4c1b9d9425bb385b634ff875406: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:39.355543 20860 tablet_bootstrap.cc:492] T bd731cd6c08949a092d51b9232397c41 P 9311f4c1b9d9425bb385b634ff875406: No bootstrap required, opened a new log
I20260925 02:56:39.355930 20860 ts_tablet_manager.cc:1397] T bd731cd6c08949a092d51b9232397c41 P 9311f4c1b9d9425bb385b634ff875406: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260925 02:56:39.358131 20859 raft_consensus.cc:348] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } } peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } }
I20260925 02:56:39.358480 20859 raft_consensus.cc:374] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:39.358590 20859 raft_consensus.cc:729] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f61ee1fa6446f19d3a807a7ca82a42, State: Initialized, Role: FOLLOWER
I20260925 02:56:39.358338 20858 raft_consensus.cc:348] T bd731cd6c08949a092d51b9232397c41 P 052b7e54f5604cde84334d328a43518a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } } peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } }
I20260925 02:56:39.358723 20858 raft_consensus.cc:374] T bd731cd6c08949a092d51b9232397c41 P 052b7e54f5604cde84334d328a43518a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:39.358903 20858 raft_consensus.cc:729] T bd731cd6c08949a092d51b9232397c41 P 052b7e54f5604cde84334d328a43518a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 052b7e54f5604cde84334d328a43518a, State: Initialized, Role: FOLLOWER
I20260925 02:56:39.359259 20859 consensus_queue.cc:260] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } } peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } }
I20260925 02:56:39.359573 20858 consensus_queue.cc:260] T bd731cd6c08949a092d51b9232397c41 P 052b7e54f5604cde84334d328a43518a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } } peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } }
I20260925 02:56:39.360307 20700 heartbeater.cc:503] Master 127.18.196.254:34431 was elected leader, sending a full tablet report...
I20260925 02:56:39.362294 20859 ts_tablet_manager.cc:1428] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42: Time spent starting tablet: real 0.012s user 0.000s sys 0.012s
I20260925 02:56:39.367892 20858 ts_tablet_manager.cc:1428] T bd731cd6c08949a092d51b9232397c41 P 052b7e54f5604cde84334d328a43518a: Time spent starting tablet: real 0.021s user 0.014s sys 0.006s
I20260925 02:56:39.368505 20860 raft_consensus.cc:348] T bd731cd6c08949a092d51b9232397c41 P 9311f4c1b9d9425bb385b634ff875406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } } peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } }
I20260925 02:56:39.368826 20860 raft_consensus.cc:374] T bd731cd6c08949a092d51b9232397c41 P 9311f4c1b9d9425bb385b634ff875406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:39.368924 20860 raft_consensus.cc:729] T bd731cd6c08949a092d51b9232397c41 P 9311f4c1b9d9425bb385b634ff875406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9311f4c1b9d9425bb385b634ff875406, State: Initialized, Role: FOLLOWER
I20260925 02:56:39.369499 20860 consensus_queue.cc:260] T bd731cd6c08949a092d51b9232397c41 P 9311f4c1b9d9425bb385b634ff875406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } } peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } }
I20260925 02:56:39.370569 20839 heartbeater.cc:503] Master 127.18.196.254:34431 was elected leader, sending a full tablet report...
I20260925 02:56:39.371156 20860 ts_tablet_manager.cc:1428] T bd731cd6c08949a092d51b9232397c41 P 9311f4c1b9d9425bb385b634ff875406: Time spent starting tablet: real 0.015s user 0.002s sys 0.014s
W20260925 02:56:39.467969 20840 tablet.cc:2369] T bd731cd6c08949a092d51b9232397c41 P 9311f4c1b9d9425bb385b634ff875406: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:56:39.480168 20701 tablet.cc:2369] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:56:39.485718 20560 tablet.cc:2369] T bd731cd6c08949a092d51b9232397c41 P 052b7e54f5604cde84334d328a43518a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:56:39.639963 20864 raft_consensus.cc:482] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:56:39.640273 20864 raft_consensus.cc:504] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } } peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } }
I20260925 02:56:39.641968 20864 leader_election.cc:288] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9311f4c1b9d9425bb385b634ff875406 (127.18.196.195:38855), 052b7e54f5604cde84334d328a43518a (127.18.196.193:42561)
I20260925 02:56:39.649585 20856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:39.642472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:33675 (local address 127.18.196.193:42561)
0925 02:56:39.642663 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:39.642669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:39.644041 (+ 1372us) server_negotiation.cc:407] Connection header received
0925 02:56:39.644325 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:39.644331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:39.644441 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:39.644534 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:39.645176 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.645911 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.647879 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.648627 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.648814 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:39.648892 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:39.649201 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 02:56:39.649325 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:56:39.649681 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:39.642472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:51977 (local address 127.18.196.195:38855)
0925 02:56:39.642658 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:39.642664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:39.644135 (+ 1471us) server_negotiation.cc:407] Connection header received
0925 02:56:39.644334 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:39.644339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:39.644441 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:39.644524 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:39.645244 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.645960 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.647998 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.648723 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.648928 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:39.649006 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:39.649297 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 02:56:39.649416 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:56:39.649765 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:39.642255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:42561 (local address 127.18.196.194:33675)
0925 02:56:39.643921 (+ 1666us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:39.643937 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:39.644165 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:39.644570 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:39.644575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:39.644618 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:39.645020 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:39.645033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.646042 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.646050 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:39.647726 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:39.647742 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.648850 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.648855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:39.649047 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:39.649070 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:56:39.649174 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 02:56:39.649304 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1446,"mutex_wait_us":23,"thread_start_us":1206,"threads_started":1}
I20260925 02:56:39.649820 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:39.642494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:38855 (local address 127.18.196.194:51977)
0925 02:56:39.644030 (+ 1536us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:39.644049 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:39.644203 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:39.644539 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:39.644544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:39.644567 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:39.645090 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:39.645100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.646108 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.646113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:39.647846 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:39.647858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.648980 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.648986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:39.649140 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:39.649161 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 02:56:39.649260 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 02:56:39.649376 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1366,"mutex_wait_us":1,"thread_start_us":869,"threads_started":1}
I20260925 02:56:39.650964 20794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd731cd6c08949a092d51b9232397c41" candidate_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9311f4c1b9d9425bb385b634ff875406" is_pre_election: true
I20260925 02:56:39.650986 20514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd731cd6c08949a092d51b9232397c41" candidate_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "052b7e54f5604cde84334d328a43518a" is_pre_election: true
I20260925 02:56:39.651480 20794 raft_consensus.cc:2454] T bd731cd6c08949a092d51b9232397c41 P 9311f4c1b9d9425bb385b634ff875406 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66f61ee1fa6446f19d3a807a7ca82a42 in term 0.
I20260925 02:56:39.651480 20514 raft_consensus.cc:2454] T bd731cd6c08949a092d51b9232397c41 P 052b7e54f5604cde84334d328a43518a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66f61ee1fa6446f19d3a807a7ca82a42 in term 0.
I20260925 02:56:39.652174 20581 leader_election.cc:302] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 052b7e54f5604cde84334d328a43518a, 66f61ee1fa6446f19d3a807a7ca82a42; no voters:
I20260925 02:56:39.652604 20864 raft_consensus.cc:2781] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:56:39.652802 20864 raft_consensus.cc:482] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:56:39.652908 20864 raft_consensus.cc:3037] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:39.655262 20864 raft_consensus.cc:504] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } } peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } }
I20260925 02:56:39.655886 20864 leader_election.cc:288] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42 [CANDIDATE]: Term 1 election: Requested vote from peers 9311f4c1b9d9425bb385b634ff875406 (127.18.196.195:38855), 052b7e54f5604cde84334d328a43518a (127.18.196.193:42561)
I20260925 02:56:39.656358 20514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd731cd6c08949a092d51b9232397c41" candidate_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "052b7e54f5604cde84334d328a43518a"
I20260925 02:56:39.656369 20794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd731cd6c08949a092d51b9232397c41" candidate_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9311f4c1b9d9425bb385b634ff875406"
I20260925 02:56:39.656574 20794 raft_consensus.cc:3037] T bd731cd6c08949a092d51b9232397c41 P 9311f4c1b9d9425bb385b634ff875406 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:39.656651 20514 raft_consensus.cc:3037] T bd731cd6c08949a092d51b9232397c41 P 052b7e54f5604cde84334d328a43518a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:39.659134 20514 raft_consensus.cc:2454] T bd731cd6c08949a092d51b9232397c41 P 052b7e54f5604cde84334d328a43518a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f61ee1fa6446f19d3a807a7ca82a42 in term 1.
I20260925 02:56:39.659328 20794 raft_consensus.cc:2454] T bd731cd6c08949a092d51b9232397c41 P 9311f4c1b9d9425bb385b634ff875406 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f61ee1fa6446f19d3a807a7ca82a42 in term 1.
I20260925 02:56:39.659703 20580 leader_election.cc:302] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66f61ee1fa6446f19d3a807a7ca82a42, 9311f4c1b9d9425bb385b634ff875406; no voters:
I20260925 02:56:39.660113 20864 raft_consensus.cc:2781] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:39.660681 20864 raft_consensus.cc:686] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42 [term 1 LEADER]: Becoming Leader. State: Replica: 66f61ee1fa6446f19d3a807a7ca82a42, State: Running, Role: LEADER
I20260925 02:56:39.661190 20864 consensus_queue.cc:237] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } } peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } }
I20260925 02:56:39.665834 20350 catalog_manager.cc:5495] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66f61ee1fa6446f19d3a807a7ca82a42 (127.18.196.194). New cstate: current_term: 1 leader_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } health_report { overall_health: HEALTHY } } }
I20260925 02:56:39.771045 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:39.766240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40876 (local address 127.18.196.194:40003)
0925 02:56:39.766400 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:39.766403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:39.766426 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:56:39.766599 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:39.766602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:39.766669 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:39.766772 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:39.767099 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.767736 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.768968 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.769479 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.769614 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:39.770022 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:39.770471 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:39.770594 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:39.770672 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 02:56:39.770732 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:56:39.778736 20514 raft_consensus.cc:1261] T bd731cd6c08949a092d51b9232397c41 P 052b7e54f5604cde84334d328a43518a [term 1 FOLLOWER]: Refusing update from remote peer 66f61ee1fa6446f19d3a807a7ca82a42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:39.778740 20794 raft_consensus.cc:1261] T bd731cd6c08949a092d51b9232397c41 P 9311f4c1b9d9425bb385b634ff875406 [term 1 FOLLOWER]: Refusing update from remote peer 66f61ee1fa6446f19d3a807a7ca82a42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:39.779631 20869 consensus_queue.cc:1035] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:39.781919 20869 consensus_queue.cc:1035] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:39.788885 20872 mvcc.cc:205] Tried to move back new op lower bound from 7333089279084744704 to 7333089278616088576. Current Snapshot: MvccSnapshot[applied={T|T < 7333089279084744704}]
I20260925 02:56:39.789917 20873 mvcc.cc:205] Tried to move back new op lower bound from 7333089279084744704 to 7333089278616088576. Current Snapshot: MvccSnapshot[applied={T|T < 7333089279084744704}]
I20260925 02:56:39.806620 20350 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:39.806885 20350 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:39.811592 20350 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73: Sending DeleteTablet for 3 replicas of tablet bd731cd6c08949a092d51b9232397c41
I20260925 02:56:39.812777 20769 tablet_service.cc:1514] Processing DeleteTablet for tablet bd731cd6c08949a092d51b9232397c41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:39 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:47608
I20260925 02:56:39.812883 20629 tablet_service.cc:1514] Processing DeleteTablet for tablet bd731cd6c08949a092d51b9232397c41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:39 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:40864
I20260925 02:56:39.812886 20489 tablet_service.cc:1514] Processing DeleteTablet for tablet bd731cd6c08949a092d51b9232397c41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:39 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:32900
I20260925 02:56:39.817358 20350 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:56:39.818150 20350 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:56:39.823755 20882 tablet_replica.cc:330] T bd731cd6c08949a092d51b9232397c41 P 9311f4c1b9d9425bb385b634ff875406: stopping tablet replica
I20260925 02:56:39.823962 20883 tablet_replica.cc:330] T bd731cd6c08949a092d51b9232397c41 P 052b7e54f5604cde84334d328a43518a: stopping tablet replica
I20260925 02:56:39.824183 20882 raft_consensus.cc:2229] T bd731cd6c08949a092d51b9232397c41 P 9311f4c1b9d9425bb385b634ff875406 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:39.824646 20882 raft_consensus.cc:2258] T bd731cd6c08949a092d51b9232397c41 P 9311f4c1b9d9425bb385b634ff875406 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:39.826857 20882 ts_tablet_manager.cc:1905] T bd731cd6c08949a092d51b9232397c41 P 9311f4c1b9d9425bb385b634ff875406: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:39.830705 20489 tablet_service.cc:1467] Processing CreateTablet for tablet b81b3e92a39e4e2da4907cebe8d29bba (DEFAULT_TABLE table=test-table [id=2f3313c6e0ac4731850bb0a5412815fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:39.831187 20489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b81b3e92a39e4e2da4907cebe8d29bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:39.831238 20769 tablet_service.cc:1467] Processing CreateTablet for tablet b81b3e92a39e4e2da4907cebe8d29bba (DEFAULT_TABLE table=test-table [id=2f3313c6e0ac4731850bb0a5412815fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:39.831732 20769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b81b3e92a39e4e2da4907cebe8d29bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:39.837421 20858 tablet_bootstrap.cc:492] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a: Bootstrap starting.
I20260925 02:56:39.837957 20628 tablet_service.cc:1467] Processing CreateTablet for tablet b81b3e92a39e4e2da4907cebe8d29bba (DEFAULT_TABLE table=test-table [id=2f3313c6e0ac4731850bb0a5412815fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:39.838723 20628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b81b3e92a39e4e2da4907cebe8d29bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:39.839241 20858 tablet_bootstrap.cc:654] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:39.844807 20860 tablet_bootstrap.cc:492] T b81b3e92a39e4e2da4907cebe8d29bba P 9311f4c1b9d9425bb385b634ff875406: Bootstrap starting.
I20260925 02:56:39.845657 20883 raft_consensus.cc:2229] T bd731cd6c08949a092d51b9232397c41 P 052b7e54f5604cde84334d328a43518a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:39.845782 20881 tablet_replica.cc:330] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42: stopping tablet replica
I20260925 02:56:39.846285 20883 raft_consensus.cc:2258] T bd731cd6c08949a092d51b9232397c41 P 052b7e54f5604cde84334d328a43518a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:39.846517 20860 tablet_bootstrap.cc:654] T b81b3e92a39e4e2da4907cebe8d29bba P 9311f4c1b9d9425bb385b634ff875406: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:39.846220 20881 raft_consensus.cc:2229] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:56:39.848059 20881 raft_consensus.cc:2258] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:39.848483 20883 ts_tablet_manager.cc:1905] T bd731cd6c08949a092d51b9232397c41 P 052b7e54f5604cde84334d328a43518a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:39.852469 20859 tablet_bootstrap.cc:492] T b81b3e92a39e4e2da4907cebe8d29bba P 66f61ee1fa6446f19d3a807a7ca82a42: Bootstrap starting.
I20260925 02:56:39.854173 20859 tablet_bootstrap.cc:654] T b81b3e92a39e4e2da4907cebe8d29bba P 66f61ee1fa6446f19d3a807a7ca82a42: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:39.856253 20883 ts_tablet_manager.cc:1918] T bd731cd6c08949a092d51b9232397c41 P 052b7e54f5604cde84334d328a43518a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:56:39.856429 20883 log.cc:1196] T bd731cd6c08949a092d51b9232397c41 P 052b7e54f5604cde84334d328a43518a: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-0/wal/wals/bd731cd6c08949a092d51b9232397c41
I20260925 02:56:39.857049 20883 ts_tablet_manager.cc:1939] T bd731cd6c08949a092d51b9232397c41 P 052b7e54f5604cde84334d328a43518a: Deleting consensus metadata
I20260925 02:56:39.858381 20882 ts_tablet_manager.cc:1918] T bd731cd6c08949a092d51b9232397c41 P 9311f4c1b9d9425bb385b634ff875406: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:56:39.858546 20882 log.cc:1196] T bd731cd6c08949a092d51b9232397c41 P 9311f4c1b9d9425bb385b634ff875406: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/bd731cd6c08949a092d51b9232397c41
I20260925 02:56:39.859508 20881 ts_tablet_manager.cc:1905] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:39.863174 20882 ts_tablet_manager.cc:1939] T bd731cd6c08949a092d51b9232397c41 P 9311f4c1b9d9425bb385b634ff875406: Deleting consensus metadata
I20260925 02:56:39.863626 20859 tablet_bootstrap.cc:492] T b81b3e92a39e4e2da4907cebe8d29bba P 66f61ee1fa6446f19d3a807a7ca82a42: No bootstrap required, opened a new log
I20260925 02:56:39.863787 20859 ts_tablet_manager.cc:1397] T b81b3e92a39e4e2da4907cebe8d29bba P 66f61ee1fa6446f19d3a807a7ca82a42: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20260925 02:56:39.858736 20337 catalog_manager.cc:4841] TS 052b7e54f5604cde84334d328a43518a (127.18.196.193:42561): tablet bd731cd6c08949a092d51b9232397c41 (table test-table [id=fa3c2423f4c042d1b50f2f94a2679936]) successfully deleted
I20260925 02:56:39.864847 20336 catalog_manager.cc:4841] TS 9311f4c1b9d9425bb385b634ff875406 (127.18.196.195:38855): tablet bd731cd6c08949a092d51b9232397c41 (table test-table [id=fa3c2423f4c042d1b50f2f94a2679936]) successfully deleted
I20260925 02:56:39.864773 20859 raft_consensus.cc:348] T b81b3e92a39e4e2da4907cebe8d29bba P 66f61ee1fa6446f19d3a807a7ca82a42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } }
I20260925 02:56:39.864990 20859 raft_consensus.cc:374] T b81b3e92a39e4e2da4907cebe8d29bba P 66f61ee1fa6446f19d3a807a7ca82a42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:39.865096 20859 raft_consensus.cc:729] T b81b3e92a39e4e2da4907cebe8d29bba P 66f61ee1fa6446f19d3a807a7ca82a42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f61ee1fa6446f19d3a807a7ca82a42, State: Initialized, Role: FOLLOWER
I20260925 02:56:39.865371 20859 consensus_queue.cc:260] T b81b3e92a39e4e2da4907cebe8d29bba P 66f61ee1fa6446f19d3a807a7ca82a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } }
I20260925 02:56:39.866153 20860 tablet_bootstrap.cc:492] T b81b3e92a39e4e2da4907cebe8d29bba P 9311f4c1b9d9425bb385b634ff875406: No bootstrap required, opened a new log
I20260925 02:56:39.866289 20860 ts_tablet_manager.cc:1397] T b81b3e92a39e4e2da4907cebe8d29bba P 9311f4c1b9d9425bb385b634ff875406: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.001s
I20260925 02:56:39.867038 20860 raft_consensus.cc:348] T b81b3e92a39e4e2da4907cebe8d29bba P 9311f4c1b9d9425bb385b634ff875406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } }
I20260925 02:56:39.867254 20860 raft_consensus.cc:374] T b81b3e92a39e4e2da4907cebe8d29bba P 9311f4c1b9d9425bb385b634ff875406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:39.867352 20860 raft_consensus.cc:729] T b81b3e92a39e4e2da4907cebe8d29bba P 9311f4c1b9d9425bb385b634ff875406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9311f4c1b9d9425bb385b634ff875406, State: Initialized, Role: FOLLOWER
I20260925 02:56:39.867713 20860 consensus_queue.cc:260] T b81b3e92a39e4e2da4907cebe8d29bba P 9311f4c1b9d9425bb385b634ff875406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } }
I20260925 02:56:39.868438 20859 ts_tablet_manager.cc:1428] T b81b3e92a39e4e2da4907cebe8d29bba P 66f61ee1fa6446f19d3a807a7ca82a42: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260925 02:56:39.869158 20860 ts_tablet_manager.cc:1428] T b81b3e92a39e4e2da4907cebe8d29bba P 9311f4c1b9d9425bb385b634ff875406: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:56:39.869205 20858 tablet_bootstrap.cc:492] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a: No bootstrap required, opened a new log
I20260925 02:56:39.869453 20858 ts_tablet_manager.cc:1397] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.009s
I20260925 02:56:39.870345 20858 raft_consensus.cc:348] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } }
I20260925 02:56:39.870682 20858 raft_consensus.cc:374] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:39.870814 20858 raft_consensus.cc:729] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 052b7e54f5604cde84334d328a43518a, State: Initialized, Role: FOLLOWER
I20260925 02:56:39.871093 20881 ts_tablet_manager.cc:1918] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:56:39.871237 20881 log.cc:1196] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/bd731cd6c08949a092d51b9232397c41
I20260925 02:56:39.871454 20858 consensus_queue.cc:260] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } }
I20260925 02:56:39.871865 20881 ts_tablet_manager.cc:1939] T bd731cd6c08949a092d51b9232397c41 P 66f61ee1fa6446f19d3a807a7ca82a42: Deleting consensus metadata
I20260925 02:56:39.872709 20858 ts_tablet_manager.cc:1428] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:56:39.873607 20335 catalog_manager.cc:4841] TS 66f61ee1fa6446f19d3a807a7ca82a42 (127.18.196.194:40003): tablet bd731cd6c08949a092d51b9232397c41 (table test-table [id=fa3c2423f4c042d1b50f2f94a2679936]) successfully deleted
I20260925 02:56:40.071108 20865 raft_consensus.cc:482] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:56:40.071348 20865 raft_consensus.cc:504] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } }
I20260925 02:56:40.073173 20865 leader_election.cc:288] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66f61ee1fa6446f19d3a807a7ca82a42 (127.18.196.194:40003), 9311f4c1b9d9425bb385b634ff875406 (127.18.196.195:38855)
I20260925 02:56:40.082104 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:40.073993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:47551 (local address 127.18.196.195:38855)
0925 02:56:40.074243 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:40.074249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:40.075274 (+ 1025us) server_negotiation.cc:407] Connection header received
0925 02:56:40.075588 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:40.075593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:40.075704 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:40.075810 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:40.076443 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.077518 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.079577 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.080915 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.081069 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:40.081159 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:40.081649 (+ 490us) server_negotiation.cc:299] Negotiation successful
0925 02:56:40.081794 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260925 02:56:40.082634 20888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:40.074446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:38855 (local address 127.18.196.193:47551)
0925 02:56:40.075154 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:40.075172 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:40.075369 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:40.075857 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:40.075862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:40.075894 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:40.076249 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:40.076262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.077643 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.077649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:40.079386 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:40.079399 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.081155 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.081163 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:40.081443 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:40.081485 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 02:56:40.081617 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 02:56:40.081765 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":37,"thread_start_us":425,"threads_started":1}
I20260925 02:56:40.083040 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:40.073891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:36121 (local address 127.18.196.194:40003)
0925 02:56:40.074131 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:40.074155 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:40.074427 (+ 272us) server_negotiation.cc:407] Connection header received
0925 02:56:40.074697 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:40.074706 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:40.074857 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:40.074937 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:40.076020 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.077131 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.080135 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.081641 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.081795 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:40.081869 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:40.082493 (+ 624us) server_negotiation.cc:299] Negotiation successful
0925 02:56:40.082615 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 02:56:40.083102 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:40.073430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:40003 (local address 127.18.196.193:36121)
0925 02:56:40.074293 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:40.074315 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:40.074487 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:40.075173 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:40.075176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:40.075200 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:40.075826 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:40.075841 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.077299 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.077304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:40.079942 (+ 2638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:40.079958 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.082109 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.082119 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:40.082299 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:40.082331 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 02:56:40.082456 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 02:56:40.082590 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":24,"thread_start_us":751,"threads_started":1}
I20260925 02:56:40.084025 20653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b81b3e92a39e4e2da4907cebe8d29bba" candidate_uuid: "052b7e54f5604cde84334d328a43518a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" is_pre_election: true
I20260925 02:56:40.084108 20794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b81b3e92a39e4e2da4907cebe8d29bba" candidate_uuid: "052b7e54f5604cde84334d328a43518a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9311f4c1b9d9425bb385b634ff875406" is_pre_election: true
I20260925 02:56:40.084379 20794 raft_consensus.cc:2454] T b81b3e92a39e4e2da4907cebe8d29bba P 9311f4c1b9d9425bb385b634ff875406 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 052b7e54f5604cde84334d328a43518a in term 0.
I20260925 02:56:40.084633 20653 raft_consensus.cc:2454] T b81b3e92a39e4e2da4907cebe8d29bba P 66f61ee1fa6446f19d3a807a7ca82a42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 052b7e54f5604cde84334d328a43518a in term 0.
I20260925 02:56:40.084926 20440 leader_election.cc:302] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 052b7e54f5604cde84334d328a43518a, 9311f4c1b9d9425bb385b634ff875406; no voters:
I20260925 02:56:40.085429 20865 raft_consensus.cc:2781] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:56:40.085572 20865 raft_consensus.cc:482] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:56:40.085678 20865 raft_consensus.cc:3037] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:40.089038 20865 raft_consensus.cc:504] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } }
I20260925 02:56:40.089758 20865 leader_election.cc:288] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a [CANDIDATE]: Term 1 election: Requested vote from peers 66f61ee1fa6446f19d3a807a7ca82a42 (127.18.196.194:40003), 9311f4c1b9d9425bb385b634ff875406 (127.18.196.195:38855)
I20260925 02:56:40.090274 20653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b81b3e92a39e4e2da4907cebe8d29bba" candidate_uuid: "052b7e54f5604cde84334d328a43518a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f61ee1fa6446f19d3a807a7ca82a42"
I20260925 02:56:40.090363 20794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b81b3e92a39e4e2da4907cebe8d29bba" candidate_uuid: "052b7e54f5604cde84334d328a43518a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9311f4c1b9d9425bb385b634ff875406"
I20260925 02:56:40.090493 20653 raft_consensus.cc:3037] T b81b3e92a39e4e2da4907cebe8d29bba P 66f61ee1fa6446f19d3a807a7ca82a42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:40.090615 20794 raft_consensus.cc:3037] T b81b3e92a39e4e2da4907cebe8d29bba P 9311f4c1b9d9425bb385b634ff875406 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:40.092927 20794 raft_consensus.cc:2454] T b81b3e92a39e4e2da4907cebe8d29bba P 9311f4c1b9d9425bb385b634ff875406 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 052b7e54f5604cde84334d328a43518a in term 1.
I20260925 02:56:40.093381 20440 leader_election.cc:302] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 052b7e54f5604cde84334d328a43518a, 9311f4c1b9d9425bb385b634ff875406; no voters:
I20260925 02:56:40.093662 20865 raft_consensus.cc:2781] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:40.094002 20653 raft_consensus.cc:2454] T b81b3e92a39e4e2da4907cebe8d29bba P 66f61ee1fa6446f19d3a807a7ca82a42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 052b7e54f5604cde84334d328a43518a in term 1.
I20260925 02:56:40.094408 20865 raft_consensus.cc:686] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a [term 1 LEADER]: Becoming Leader. State: Replica: 052b7e54f5604cde84334d328a43518a, State: Running, Role: LEADER
I20260925 02:56:40.095184 20865 consensus_queue.cc:237] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } }
I20260925 02:56:40.100706 20351 catalog_manager.cc:5495] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a reported cstate change: term changed from 0 to 1, leader changed from <none> to 052b7e54f5604cde84334d328a43518a (127.18.196.193). New cstate: current_term: 1 leader_uuid: "052b7e54f5604cde84334d328a43518a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } health_report { overall_health: UNKNOWN } } }
Sep 25 02:56:40 dist-test-slave-8zvk krb5kdc[20306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304997, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:40.172681 20897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:40.157385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58062 (local address 127.18.196.254:34431)
0925 02:56:40.158337 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:40.158345 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:40.158369 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:56:40.158422 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:40.158425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:40.158480 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:40.158589 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:40.159780 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.160586 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.161851 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.162383 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.162531 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:40.166687 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:40.166692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:40.166695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:40.166724 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:40.169554 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:40.170589 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:40.170593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:40.170596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:40.170716 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:40.171242 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:40.171245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:40.171247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:40.171503 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:40.171609 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:40.172229 (+ 620us) server_negotiation.cc:299] Negotiation successful
0925 02:56:40.172396 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":838,"thread_start_us":679,"threads_started":1}
I20260925 02:56:40.173247 20896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:40.157368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:34431 (local address 127.0.0.1:58062)
0925 02:56:40.158078 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:40.158102 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:40.158259 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:40.158618 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:40.158624 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:40.159377 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:40.159598 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:40.159608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.160712 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.160720 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:40.161707 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:40.161719 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.162551 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.162557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:40.162745 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:40.163479 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:40.163497 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:40.166378 (+ 2881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:40.169695 (+ 3317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:40.169708 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:40.169716 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:40.170461 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:40.170870 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:40.170878 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:40.170881 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:40.171099 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:40.171640 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:40.171648 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:40.171892 (+ 244us) client_negotiation.cc:769] Sending connection context
0925 02:56:40.172167 (+ 275us) client_negotiation.cc:240] Negotiation successful
0925 02:56:40.172635 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":74,"thread_start_us":364,"threads_started":1}
I20260925 02:56:40.178756 20351 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:58062:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 02:56:40.179932 20351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:56:40.187223 20489 tablet_service.cc:1467] Processing CreateTablet for tablet 09e8b6801496480897bfa424169807c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=35b77ddb64304d8baebe582f22310119]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:56:40.187283 20628 tablet_service.cc:1467] Processing CreateTablet for tablet 09e8b6801496480897bfa424169807c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=35b77ddb64304d8baebe582f22310119]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:56:40.187827 20628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09e8b6801496480897bfa424169807c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:40.187846 20489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09e8b6801496480897bfa424169807c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:40.191787 20769 tablet_service.cc:1467] Processing CreateTablet for tablet 09e8b6801496480897bfa424169807c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=35b77ddb64304d8baebe582f22310119]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:56:40.192373 20769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09e8b6801496480897bfa424169807c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:40.195158 20858 tablet_bootstrap.cc:492] T 09e8b6801496480897bfa424169807c6 P 052b7e54f5604cde84334d328a43518a: Bootstrap starting.
I20260925 02:56:40.196897 20858 tablet_bootstrap.cc:654] T 09e8b6801496480897bfa424169807c6 P 052b7e54f5604cde84334d328a43518a: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:40.198117 20859 tablet_bootstrap.cc:492] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42: Bootstrap starting.
I20260925 02:56:40.200083 20859 tablet_bootstrap.cc:654] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:40.200678 20860 tablet_bootstrap.cc:492] T 09e8b6801496480897bfa424169807c6 P 9311f4c1b9d9425bb385b634ff875406: Bootstrap starting.
I20260925 02:56:40.202605 20860 tablet_bootstrap.cc:654] T 09e8b6801496480897bfa424169807c6 P 9311f4c1b9d9425bb385b634ff875406: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:40.203195 20858 tablet_bootstrap.cc:492] T 09e8b6801496480897bfa424169807c6 P 052b7e54f5604cde84334d328a43518a: No bootstrap required, opened a new log
I20260925 02:56:40.203364 20858 ts_tablet_manager.cc:1397] T 09e8b6801496480897bfa424169807c6 P 052b7e54f5604cde84334d328a43518a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260925 02:56:40.203728 20859 tablet_bootstrap.cc:492] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42: No bootstrap required, opened a new log
I20260925 02:56:40.204010 20859 ts_tablet_manager.cc:1397] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 02:56:40.204237 20858 raft_consensus.cc:348] T 09e8b6801496480897bfa424169807c6 P 052b7e54f5604cde84334d328a43518a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } }
I20260925 02:56:40.204876 20858 raft_consensus.cc:374] T 09e8b6801496480897bfa424169807c6 P 052b7e54f5604cde84334d328a43518a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:40.204983 20858 raft_consensus.cc:729] T 09e8b6801496480897bfa424169807c6 P 052b7e54f5604cde84334d328a43518a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 052b7e54f5604cde84334d328a43518a, State: Initialized, Role: FOLLOWER
I20260925 02:56:40.205348 20858 consensus_queue.cc:260] T 09e8b6801496480897bfa424169807c6 P 052b7e54f5604cde84334d328a43518a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } }
I20260925 02:56:40.205796 20865 tablet_replica.cc:439] T 09e8b6801496480897bfa424169807c6 P 052b7e54f5604cde84334d328a43518a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } } }
I20260925 02:56:40.205828 20859 raft_consensus.cc:348] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } }
I20260925 02:56:40.206102 20865 tablet_replica.cc:442] T 09e8b6801496480897bfa424169807c6 P 052b7e54f5604cde84334d328a43518a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:56:40.206127 20859 raft_consensus.cc:374] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:40.206235 20859 raft_consensus.cc:729] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f61ee1fa6446f19d3a807a7ca82a42, State: Initialized, Role: FOLLOWER
I20260925 02:56:40.206975 20859 consensus_queue.cc:260] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } }
I20260925 02:56:40.207429 20864 tablet_replica.cc:439] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } } }
I20260925 02:56:40.207741 20864 tablet_replica.cc:442] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:56:40.209092 20858 ts_tablet_manager.cc:1428] T 09e8b6801496480897bfa424169807c6 P 052b7e54f5604cde84334d328a43518a: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260925 02:56:40.211570 20859 ts_tablet_manager.cc:1428] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260925 02:56:40.212311 20860 tablet_bootstrap.cc:492] T 09e8b6801496480897bfa424169807c6 P 9311f4c1b9d9425bb385b634ff875406: No bootstrap required, opened a new log
I20260925 02:56:40.212566 20860 ts_tablet_manager.cc:1397] T 09e8b6801496480897bfa424169807c6 P 9311f4c1b9d9425bb385b634ff875406: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260925 02:56:40.213404 20860 raft_consensus.cc:348] T 09e8b6801496480897bfa424169807c6 P 9311f4c1b9d9425bb385b634ff875406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } }
I20260925 02:56:40.213665 20860 raft_consensus.cc:374] T 09e8b6801496480897bfa424169807c6 P 9311f4c1b9d9425bb385b634ff875406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:40.213770 20860 raft_consensus.cc:729] T 09e8b6801496480897bfa424169807c6 P 9311f4c1b9d9425bb385b634ff875406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9311f4c1b9d9425bb385b634ff875406, State: Initialized, Role: FOLLOWER
I20260925 02:56:40.214092 20860 consensus_queue.cc:260] T 09e8b6801496480897bfa424169807c6 P 9311f4c1b9d9425bb385b634ff875406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } }
I20260925 02:56:40.214848 20866 tablet_replica.cc:439] T 09e8b6801496480897bfa424169807c6 P 9311f4c1b9d9425bb385b634ff875406: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } } }
I20260925 02:56:40.215193 20866 tablet_replica.cc:442] T 09e8b6801496480897bfa424169807c6 P 9311f4c1b9d9425bb385b634ff875406: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:56:40.217573 20860 ts_tablet_manager.cc:1428] T 09e8b6801496480897bfa424169807c6 P 9311f4c1b9d9425bb385b634ff875406: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260925 02:56:40.314417 20864 raft_consensus.cc:482] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:56:40.314651 20864 raft_consensus.cc:504] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } }
I20260925 02:56:40.315249 20864 leader_election.cc:288] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 052b7e54f5604cde84334d328a43518a (127.18.196.193:42561), 9311f4c1b9d9425bb385b634ff875406 (127.18.196.195:38855)
I20260925 02:56:40.315848 20514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09e8b6801496480897bfa424169807c6" candidate_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "052b7e54f5604cde84334d328a43518a" is_pre_election: true
I20260925 02:56:40.315886 20794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09e8b6801496480897bfa424169807c6" candidate_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9311f4c1b9d9425bb385b634ff875406" is_pre_election: true
I20260925 02:56:40.316109 20514 raft_consensus.cc:2454] T 09e8b6801496480897bfa424169807c6 P 052b7e54f5604cde84334d328a43518a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66f61ee1fa6446f19d3a807a7ca82a42 in term 0.
I20260925 02:56:40.316164 20794 raft_consensus.cc:2454] T 09e8b6801496480897bfa424169807c6 P 9311f4c1b9d9425bb385b634ff875406 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66f61ee1fa6446f19d3a807a7ca82a42 in term 0.
I20260925 02:56:40.316551 20581 leader_election.cc:302] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 052b7e54f5604cde84334d328a43518a, 66f61ee1fa6446f19d3a807a7ca82a42; no voters:
I20260925 02:56:40.316838 20864 raft_consensus.cc:2781] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:56:40.316923 20864 raft_consensus.cc:482] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:56:40.317044 20864 raft_consensus.cc:3037] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:40.319046 20864 raft_consensus.cc:504] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } }
I20260925 02:56:40.319583 20864 leader_election.cc:288] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42 [CANDIDATE]: Term 1 election: Requested vote from peers 052b7e54f5604cde84334d328a43518a (127.18.196.193:42561), 9311f4c1b9d9425bb385b634ff875406 (127.18.196.195:38855)
I20260925 02:56:40.320102 20794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09e8b6801496480897bfa424169807c6" candidate_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9311f4c1b9d9425bb385b634ff875406"
I20260925 02:56:40.320120 20514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09e8b6801496480897bfa424169807c6" candidate_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "052b7e54f5604cde84334d328a43518a"
I20260925 02:56:40.320304 20514 raft_consensus.cc:3037] T 09e8b6801496480897bfa424169807c6 P 052b7e54f5604cde84334d328a43518a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:40.320324 20794 raft_consensus.cc:3037] T 09e8b6801496480897bfa424169807c6 P 9311f4c1b9d9425bb385b634ff875406 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:40.323215 20794 raft_consensus.cc:2454] T 09e8b6801496480897bfa424169807c6 P 9311f4c1b9d9425bb385b634ff875406 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f61ee1fa6446f19d3a807a7ca82a42 in term 1.
I20260925 02:56:40.323232 20514 raft_consensus.cc:2454] T 09e8b6801496480897bfa424169807c6 P 052b7e54f5604cde84334d328a43518a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f61ee1fa6446f19d3a807a7ca82a42 in term 1.
I20260925 02:56:40.323675 20581 leader_election.cc:302] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 052b7e54f5604cde84334d328a43518a, 66f61ee1fa6446f19d3a807a7ca82a42; no voters:
I20260925 02:56:40.323915 20864 raft_consensus.cc:2781] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:40.324270 20864 raft_consensus.cc:686] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42 [term 1 LEADER]: Becoming Leader. State: Replica: 66f61ee1fa6446f19d3a807a7ca82a42, State: Running, Role: LEADER
I20260925 02:56:40.324522 20864 consensus_queue.cc:237] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } }
I20260925 02:56:40.326157 20901 tablet_replica.cc:439] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42: TxnStatusTablet state changed. Reason: New leader 66f61ee1fa6446f19d3a807a7ca82a42. Latest consensus state: current_term: 1 leader_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } } }
I20260925 02:56:40.326432 20901 tablet_replica.cc:442] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:56:40.327471 20349 catalog_manager.cc:5495] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66f61ee1fa6446f19d3a807a7ca82a42 (127.18.196.194). New cstate: current_term: 1 leader_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } health_report { overall_health: UNKNOWN } } }
I20260925 02:56:40.327816 20903 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 02:56:40.328096 20903 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 02:56:40.365574 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:40.359709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40880 (local address 127.18.196.194:40003)
0925 02:56:40.359925 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:40.359930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:40.359946 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:56:40.360159 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:40.360162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:40.360225 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:40.360306 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:40.360743 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.361549 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.363556 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.364127 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.364230 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:40.364767 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:40.364867 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:40.364936 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:40.365141 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 02:56:40.365242 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:56:40.366039 20896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:40.359523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:40003 (local address 127.0.0.1:40880)
0925 02:56:40.359805 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:40.359829 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:40.359978 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:40.360362 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:40.360368 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:40.360382 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:56:40.360595 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:40.360607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.361682 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.361688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:40.363395 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:40.363411 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.364289 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.364295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:40.364506 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:40.364581 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:56:40.364974 (+ 393us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:56:40.364995 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 02:56:40.365092 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 02:56:40.365303 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 02:56:40.370059 20514 raft_consensus.cc:1261] T 09e8b6801496480897bfa424169807c6 P 052b7e54f5604cde84334d328a43518a [term 1 FOLLOWER]: Refusing update from remote peer 66f61ee1fa6446f19d3a807a7ca82a42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:40.370162 20794 raft_consensus.cc:1261] T 09e8b6801496480897bfa424169807c6 P 9311f4c1b9d9425bb385b634ff875406 [term 1 FOLLOWER]: Refusing update from remote peer 66f61ee1fa6446f19d3a807a7ca82a42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:40.370721 20901 consensus_queue.cc:1035] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:40.371104 20864 consensus_queue.cc:1035] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:40.380362 20865 tablet_replica.cc:439] T 09e8b6801496480897bfa424169807c6 P 052b7e54f5604cde84334d328a43518a: TxnStatusTablet state changed. Reason: New leader 66f61ee1fa6446f19d3a807a7ca82a42. Latest consensus state: current_term: 1 leader_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } } }
I20260925 02:56:40.380625 20865 tablet_replica.cc:442] T 09e8b6801496480897bfa424169807c6 P 052b7e54f5604cde84334d328a43518a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:56:40.381675 20866 tablet_replica.cc:439] T 09e8b6801496480897bfa424169807c6 P 9311f4c1b9d9425bb385b634ff875406: TxnStatusTablet state changed. Reason: New leader 66f61ee1fa6446f19d3a807a7ca82a42. Latest consensus state: current_term: 1 leader_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } } }
I20260925 02:56:40.381963 20866 tablet_replica.cc:442] T 09e8b6801496480897bfa424169807c6 P 9311f4c1b9d9425bb385b634ff875406: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:56:40.387887 20907 mvcc.cc:205] Tried to move back new op lower bound from 7333089281509765120 to 7333089281331376128. Current Snapshot: MvccSnapshot[applied={T|T < 7333089281509765120}]
I20260925 02:56:40.387864 20865 tablet_replica.cc:439] T 09e8b6801496480897bfa424169807c6 P 052b7e54f5604cde84334d328a43518a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } } }
I20260925 02:56:40.388211 20865 tablet_replica.cc:442] T 09e8b6801496480897bfa424169807c6 P 052b7e54f5604cde84334d328a43518a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:56:40.389803 20866 tablet_replica.cc:439] T 09e8b6801496480897bfa424169807c6 P 9311f4c1b9d9425bb385b634ff875406: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } } }
I20260925 02:56:40.390051 20866 tablet_replica.cc:442] T 09e8b6801496480897bfa424169807c6 P 9311f4c1b9d9425bb385b634ff875406: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:56:40.395524 20910 tablet_replica.cc:439] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } } }
I20260925 02:56:40.395903 20910 tablet_replica.cc:442] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:56:40.396763 20864 tablet_replica.cc:439] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052b7e54f5604cde84334d328a43518a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42561 } } peers { permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 } } peers { permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 } } }
I20260925 02:56:40.397050 20864 tablet_replica.cc:442] T 09e8b6801496480897bfa424169807c6 P 66f61ee1fa6446f19d3a807a7ca82a42: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:56:40.414214 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:40.403472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32908 (local address 127.18.196.193:42561)
0925 02:56:40.404109 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:40.404115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:40.407966 (+ 3851us) server_negotiation.cc:407] Connection header received
0925 02:56:40.408138 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:40.408145 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:40.408257 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:40.408434 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:40.408864 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.410038 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.411500 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.412194 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.412327 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:40.412803 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:40.413523 (+ 720us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:40.413663 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:40.413763 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 02:56:40.413849 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":377,"threads_started":1}
I20260925 02:56:40.430161 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:40.424458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:41623 (local address 127.18.196.194:40003)
0925 02:56:40.424690 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:40.424696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:40.424959 (+ 263us) server_negotiation.cc:407] Connection header received
0925 02:56:40.425167 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:40.425171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:40.425236 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:40.425350 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:40.425798 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.426659 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.428112 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.428702 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.428862 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:40.429407 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:40.429553 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:40.429660 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:40.429788 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 02:56:40.429866 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:56:40.430271 20919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:40.424284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:40003 (local address 127.18.196.193:41623)
0925 02:56:40.424862 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:40.424877 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:40.425029 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:40.425352 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:40.425356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:40.425368 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:56:40.425659 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:40.425667 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.426789 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.426794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:40.427901 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:40.427913 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.428996 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.429002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:40.429148 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:40.429196 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:56:40.429659 (+ 463us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:56:40.429671 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 02:56:40.429767 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 02:56:40.429908 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":276,"threads_started":1}
I20260925 02:56:40.439211 20793 raft_consensus.cc:1261] T b81b3e92a39e4e2da4907cebe8d29bba P 9311f4c1b9d9425bb385b634ff875406 [term 1 FOLLOWER]: Refusing update from remote peer 052b7e54f5604cde84334d328a43518a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:40.439685 20653 raft_consensus.cc:1261] T b81b3e92a39e4e2da4907cebe8d29bba P 66f61ee1fa6446f19d3a807a7ca82a42 [term 1 FOLLOWER]: Refusing update from remote peer 052b7e54f5604cde84334d328a43518a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:40.440007 20889 consensus_queue.cc:1035] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9311f4c1b9d9425bb385b634ff875406" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:40.440393 20865 consensus_queue.cc:1035] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f61ee1fa6446f19d3a807a7ca82a42" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:40.490012 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:40.482682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:56333 (local address 127.18.196.193:42561)
0925 02:56:40.482872 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:40.482880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:40.483854 (+ 974us) server_negotiation.cc:407] Connection header received
0925 02:56:40.484053 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:40.484060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:40.484145 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:40.484251 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:40.485024 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.486179 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.488152 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.488626 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.488792 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:40.489294 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:40.489426 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:40.489530 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:40.489652 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 02:56:40.489735 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:56:40.490031 20924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:40.483001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:42561 (local address 127.18.196.194:56333)
0925 02:56:40.483740 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:40.483760 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:40.483922 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:40.484269 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:40.484272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:40.484281 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:56:40.484825 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:40.484835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.486389 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.486394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:40.487985 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:40.487997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.488845 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.488852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:40.488992 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:40.489040 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:56:40.489533 (+ 493us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:56:40.489542 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 02:56:40.489606 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 02:56:40.489706 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":350,"threads_started":1}
I20260925 02:56:40.535463 20349 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:40.535735 20349 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:40.539040 20349 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ef5b7ebf3814ac1ab7bdedcdce87b73: Sending DeleteTablet for 3 replicas of tablet b81b3e92a39e4e2da4907cebe8d29bba
I20260925 02:56:40.539723 20489 tablet_service.cc:1514] Processing DeleteTablet for tablet b81b3e92a39e4e2da4907cebe8d29bba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:40 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:32900
I20260925 02:56:40.540064 20769 tablet_service.cc:1514] Processing DeleteTablet for tablet b81b3e92a39e4e2da4907cebe8d29bba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:40 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:47608
I20260925 02:56:40.540671 20628 tablet_service.cc:1514] Processing DeleteTablet for tablet b81b3e92a39e4e2da4907cebe8d29bba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:40 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:40864
I20260925 02:56:40.552018 20930 tablet_replica.cc:330] T b81b3e92a39e4e2da4907cebe8d29bba P 9311f4c1b9d9425bb385b634ff875406: stopping tablet replica
I20260925 02:56:40.552220 20930 raft_consensus.cc:2229] T b81b3e92a39e4e2da4907cebe8d29bba P 9311f4c1b9d9425bb385b634ff875406 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:40.552472 20930 raft_consensus.cc:2258] T b81b3e92a39e4e2da4907cebe8d29bba P 9311f4c1b9d9425bb385b634ff875406 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:40.554663 20930 ts_tablet_manager.cc:1905] T b81b3e92a39e4e2da4907cebe8d29bba P 9311f4c1b9d9425bb385b634ff875406: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:40.560830 20929 tablet_replica.cc:330] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a: stopping tablet replica
I20260925 02:56:40.561029 20929 raft_consensus.cc:2229] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:56:40.561635 20929 raft_consensus.cc:2258] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:40.562345 20931 tablet_replica.cc:330] T b81b3e92a39e4e2da4907cebe8d29bba P 66f61ee1fa6446f19d3a807a7ca82a42: stopping tablet replica
I20260925 02:56:40.562642 20931 raft_consensus.cc:2229] T b81b3e92a39e4e2da4907cebe8d29bba P 66f61ee1fa6446f19d3a807a7ca82a42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:40.563010 20931 raft_consensus.cc:2258] T b81b3e92a39e4e2da4907cebe8d29bba P 66f61ee1fa6446f19d3a807a7ca82a42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:40.564206 20931 ts_tablet_manager.cc:1905] T b81b3e92a39e4e2da4907cebe8d29bba P 66f61ee1fa6446f19d3a807a7ca82a42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:40.566083 20929 ts_tablet_manager.cc:1905] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:40.571748 20931 ts_tablet_manager.cc:1918] T b81b3e92a39e4e2da4907cebe8d29bba P 66f61ee1fa6446f19d3a807a7ca82a42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:56:40.571946 20931 log.cc:1196] T b81b3e92a39e4e2da4907cebe8d29bba P 66f61ee1fa6446f19d3a807a7ca82a42: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/b81b3e92a39e4e2da4907cebe8d29bba
I20260925 02:56:40.572567 20931 ts_tablet_manager.cc:1939] T b81b3e92a39e4e2da4907cebe8d29bba P 66f61ee1fa6446f19d3a807a7ca82a42: Deleting consensus metadata
I20260925 02:56:40.573561 20929 ts_tablet_manager.cc:1918] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:56:40.573719 20929 log.cc:1196] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-0/wal/wals/b81b3e92a39e4e2da4907cebe8d29bba
I20260925 02:56:40.573791 20335 catalog_manager.cc:4841] TS 66f61ee1fa6446f19d3a807a7ca82a42 (127.18.196.194:40003): tablet b81b3e92a39e4e2da4907cebe8d29bba (table test-table [id=2f3313c6e0ac4731850bb0a5412815fc]) successfully deleted
I20260925 02:56:40.578742 20930 ts_tablet_manager.cc:1918] T b81b3e92a39e4e2da4907cebe8d29bba P 9311f4c1b9d9425bb385b634ff875406: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:56:40.578905 20930 log.cc:1196] T b81b3e92a39e4e2da4907cebe8d29bba P 9311f4c1b9d9425bb385b634ff875406: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/b81b3e92a39e4e2da4907cebe8d29bba
I20260925 02:56:40.579353 20930 ts_tablet_manager.cc:1939] T b81b3e92a39e4e2da4907cebe8d29bba P 9311f4c1b9d9425bb385b634ff875406: Deleting consensus metadata
Sep 25 02:56:40 dist-test-slave-8zvk krb5kdc[20306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304999, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.196.193@KRBTEST.COM
I20260925 02:56:40.580374 20336 catalog_manager.cc:4841] TS 9311f4c1b9d9425bb385b634ff875406 (127.18.196.195:38855): tablet b81b3e92a39e4e2da4907cebe8d29bba (table test-table [id=2f3313c6e0ac4731850bb0a5412815fc]) successfully deleted
I20260925 02:56:40.581239 20929 ts_tablet_manager.cc:1939] T b81b3e92a39e4e2da4907cebe8d29bba P 052b7e54f5604cde84334d328a43518a: Deleting consensus metadata
I20260925 02:56:40.582721 20337 catalog_manager.cc:4841] TS 052b7e54f5604cde84334d328a43518a (127.18.196.193:42561): tablet b81b3e92a39e4e2da4907cebe8d29bba (table test-table [id=2f3313c6e0ac4731850bb0a5412815fc]) successfully deleted
I20260925 02:56:40.589372 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:40.550415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32914 (local address 127.18.196.193:42561)
0925 02:56:40.550624 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:40.550630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:40.553167 (+ 2537us) server_negotiation.cc:407] Connection header received
0925 02:56:40.553420 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:40.553426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:40.553525 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:40.553603 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:40.561929 (+ 8326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.563124 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.574152 (+ 11028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.575433 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.575574 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:40.581457 (+ 5883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:40.581472 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:40.581490 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:40.581531 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:40.585871 (+ 4340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:40.586941 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:40.586950 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:40.586954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:40.587114 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:40.587625 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:40.587631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:40.587634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:40.588066 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:40.588358 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:40.588729 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 02:56:40.588950 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 02:56:40.590107 20554 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32914
I20260925 02:56:40.610581 20897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:40.593581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58074 (local address 127.18.196.254:34431)
0925 02:56:40.593771 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:40.593776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:40.594276 (+ 500us) server_negotiation.cc:407] Connection header received
0925 02:56:40.594448 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:40.594455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:40.594546 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:40.594647 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:40.596453 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.597719 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.599539 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.600222 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.600337 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:40.603358 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:40.603364 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:40.603369 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:40.603405 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:40.607259 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:40.608215 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:40.608223 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:40.608226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:40.608409 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:40.608907 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:40.608913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:40.608916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:40.609335 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:40.609589 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:40.610014 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 02:56:40.610248 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 02:56:40.611234 20349 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58074
I20260925 02:56:40.615499 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 20422
I20260925 02:56:40.645262 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:40.618273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.18.196.193:42561)
0925 02:56:40.618501 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:40.618507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:40.627319 (+ 8812us) server_negotiation.cc:407] Connection header received
0925 02:56:40.627549 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:40.627556 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:40.627644 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:40.627720 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:40.629664 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.630753 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.632784 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.633612 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.633730 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:40.637101 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:40.637109 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:40.637114 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:40.637180 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:40.641387 (+ 4207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:40.642302 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:40.642308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:40.642312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:40.642537 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:40.643017 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:40.643022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:40.643026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:40.643408 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:40.643793 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:40.644554 (+ 761us) server_negotiation.cc:299] Negotiation successful
0925 02:56:40.644864 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260925 02:56:40.647053 20554 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32922
W20260925 02:56:40.647696 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 20422: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:56:40.647890 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 20422
I20260925 02:56:40.668056 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 20563
Sep 25 02:56:40 dist-test-slave-8zvk krb5kdc[20306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304999, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.196.194@KRBTEST.COM
I20260925 02:56:40.683526 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:40.668880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40884 (local address 127.18.196.194:40003)
0925 02:56:40.669062 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:40.669070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:40.669093 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:56:40.669292 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:40.669297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:40.669377 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:40.669505 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:40.670665 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.671501 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.672891 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.673349 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.673487 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:40.677611 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:40.677623 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:40.677641 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:40.677676 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:40.680819 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:40.681510 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:40.681525 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:40.681527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:40.681654 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:40.682032 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:40.682037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:40.682041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:40.682403 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:40.682725 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:40.682988 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 02:56:40.683217 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 02:56:40.684165 20695 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40884
W20260925 02:56:40.684865 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 20563: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:56:40.685026 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 20563
I20260925 02:56:40.708969 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 20704
Sep 25 02:56:40 dist-test-slave-8zvk krb5kdc[20306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304999, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.196.195@KRBTEST.COM
I20260925 02:56:40.724112 20937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:40.709784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47620 (local address 127.18.196.195:38855)
0925 02:56:40.710296 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:40.710300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:40.710316 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:56:40.710369 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:40.710372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:40.710440 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:40.710535 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:40.712002 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.712804 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.714044 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.714444 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.714539 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:40.718445 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:40.718459 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:40.718475 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:40.718507 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:40.721355 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:40.722033 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:40.722040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:40.722042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:40.722168 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:40.722539 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:40.722544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:40.722547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:40.722957 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:40.723284 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:40.723549 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 02:56:40.723782 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":259,"threads_started":1}
W20260925 02:56:40.724754 20834 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47620
W20260925 02:56:40.725433 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 20704: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:56:40.725575 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 20704
I20260925 02:56:40.743650 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 20319
I20260925 02:56:40.744686 20391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:56:40.858114 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 20319
2026-09-25T02:56:40Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4940 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:40 dist-test-slave-8zvk krb5kdc[20941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:40 dist-test-slave-8zvk krb5kdc[20941](info): set up 2 sockets
Sep 25 02:56:40 dist-test-slave-8zvk krb5kdc[20941](info): commencing operation
krb5kdc: starting...
W20260925 02:56:41.942505 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 02:56:42 dist-test-slave-8zvk krb5kdc[20941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 02:56:42.126621 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:40507
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:43067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:40507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:42.435603 20954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:42.435858 20954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:42.436029 20954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:42.442970 20954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:42.443079 20954 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 02:56:42.443145 20954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:42.443251 20954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:42.443333 20954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:42.443431 20954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:42.452594 20954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:43067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:40507
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:40507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:42.453715 20954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:42.455039 20954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:42.466058 20959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:42.466092 20962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:42.466001 20960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:42.467154 20954 server_base.cc:1034] running on GCE node
I20260925 02:56:42.467839 20954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:42.469025 20954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:42.470256 20954 hybrid_clock.cc:648] HybridClock initialized: now 1790305002470201 us; error 31 us; skew 500 ppm
I20260925 02:56:42.470561 20954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:42 dist-test-slave-8zvk krb5kdc[20941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305002, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:42.473420 20954 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 02:56:42.474797 20954 webserver.cc:460] Webserver started at http://127.18.196.254:42007/ using document root <none> and password file <none>
I20260925 02:56:42.475510 20954 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:42.475636 20954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:42.475936 20954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:42.478688 20954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "1b5e1c717ad045279c8ba63b64375193"
format_stamp: "Formatted at 2026-09-25 02:56:42 on dist-test-slave-8zvk"
server_key: "b7705e8bd126f8233beb704cbd53e005"
server_key_iv: "b754c092db4c4ceeb91846fc789e9239"
server_key_version: "encryptionkey@0"
I20260925 02:56:42.479489 20954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "1b5e1c717ad045279c8ba63b64375193"
format_stamp: "Formatted at 2026-09-25 02:56:42 on dist-test-slave-8zvk"
server_key: "b7705e8bd126f8233beb704cbd53e005"
server_key_iv: "b754c092db4c4ceeb91846fc789e9239"
server_key_version: "encryptionkey@0"
I20260925 02:56:42.484193 20954 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 02:56:42.487445 20969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:42.488622 20954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:56:42.488787 20954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "1b5e1c717ad045279c8ba63b64375193"
format_stamp: "Formatted at 2026-09-25 02:56:42 on dist-test-slave-8zvk"
server_key: "b7705e8bd126f8233beb704cbd53e005"
server_key_iv: "b754c092db4c4ceeb91846fc789e9239"
server_key_version: "encryptionkey@0"
I20260925 02:56:42.488993 20954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:42.514145 20954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:42.517973 20954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:42.518236 20954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:42.539633 20954 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:40507
I20260925 02:56:42.539654 21030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:40507 every 8 connection(s)
I20260925 02:56:42.540925 20954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:42.542505 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 20954
I20260925 02:56:42.542721 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:56:42.543398 19219 external_mini_cluster.cc:1428] Setting key 9d5a74a1fb0cd20911c15a669779ca2f
I20260925 02:56:42.544986 21031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 02:56:42 dist-test-slave-8zvk krb5kdc[20941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:42.558526 21031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193: Bootstrap starting.
I20260925 02:56:42.562036 21031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:42.563374 21031 log.cc:824] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:42.566160 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:42.545320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43670 (local address 127.18.196.254:40507)
0925 02:56:42.546165 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:42.546181 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:42.546233 (+ 52us) server_negotiation.cc:407] Connection header received
0925 02:56:42.548212 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:42.548239 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:42.548760 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:42.549192 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:42.550397 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.552501 (+ 2104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.553653 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.554696 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.554909 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:42.559146 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:42.559154 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:42.559167 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:42.559204 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:42.562025 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:42.562819 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:42.562825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:42.562830 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:42.563044 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:42.563462 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:42.563466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:42.563468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:42.564154 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:42.564460 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:42.564793 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 02:56:42.565171 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":376,"threads_started":1}
I20260925 02:56:42.566411 21031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193: No bootstrap required, opened a new log
I20260925 02:56:42.573853 21031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5e1c717ad045279c8ba63b64375193" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40507 } }
I20260925 02:56:42.574198 21031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:42.574353 21031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b5e1c717ad045279c8ba63b64375193, State: Initialized, Role: FOLLOWER
I20260925 02:56:42.575014 21031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5e1c717ad045279c8ba63b64375193" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40507 } }
I20260925 02:56:42.575233 21031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:42.575366 21031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:42.575536 21031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:42.577613 21031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5e1c717ad045279c8ba63b64375193" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40507 } }
I20260925 02:56:42.578084 21031 leader_election.cc:302] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b5e1c717ad045279c8ba63b64375193; no voters:
I20260925 02:56:42.578604 21031 leader_election.cc:288] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:42.578737 21036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:42.579468 21036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193 [term 1 LEADER]: Becoming Leader. State: Replica: 1b5e1c717ad045279c8ba63b64375193, State: Running, Role: LEADER
I20260925 02:56:42.580024 21036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5e1c717ad045279c8ba63b64375193" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40507 } }
I20260925 02:56:42.580575 21031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:42.582616 21038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b5e1c717ad045279c8ba63b64375193. Latest consensus state: current_term: 1 leader_uuid: "1b5e1c717ad045279c8ba63b64375193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5e1c717ad045279c8ba63b64375193" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40507 } } }
I20260925 02:56:42.582975 21038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:42.584010 21037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b5e1c717ad045279c8ba63b64375193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5e1c717ad045279c8ba63b64375193" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40507 } } }
I20260925 02:56:42.584434 21037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:42.585969 21043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:42.591140 21043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:42.599790 21043 catalog_manager.cc:1347] Generated new cluster ID: 296cf0337f9a41349108deb00a71f840
I20260925 02:56:42.599895 21043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:42.606876 21043 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:42.608649 21043 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:42.622581 21043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193: Generated new TSK 0
I20260925 02:56:42.623478 21043 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
I20260925 02:56:42.670482 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:40507
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:43067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:42.978958 21057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:42.979190 21057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:42.979348 21057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:42.986508 21057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:42.986590 21057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:42.986737 21057 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 02:56:42.986812 21057 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:56:42.986976 21057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 02:56:42.995589 21057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:43067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.196.254:40507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:42.996591 21057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:42.997725 21057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:43.009514 21063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:43.009563 21065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:43.009644 21062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:43.010332 21057 server_base.cc:1034] running on GCE node
I20260925 02:56:43.010946 21057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:43.011755 21057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:43.012959 21057 hybrid_clock.cc:648] HybridClock initialized: now 1790305003012904 us; error 44 us; skew 500 ppm
I20260925 02:56:43.013240 21057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:43 dist-test-slave-8zvk krb5kdc[20941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305003, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:43.016173 21057 init.cc:377] Logged in from keytab as kudu/127.18.196.193@KRBTEST.COM (short username kudu)
I20260925 02:56:43.017405 21057 webserver.cc:460] Webserver started at http://127.18.196.193:40803/ using document root <none> and password file <none>
I20260925 02:56:43.017992 21057 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:43.018108 21057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:43.018414 21057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:43.020911 21057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "ef3ac5745fdc4b1b994f479bc1c69f60"
format_stamp: "Formatted at 2026-09-25 02:56:43 on dist-test-slave-8zvk"
server_key: "cc62ecbac1e4a5974d907e082f3e8a7f"
server_key_iv: "9e9bc173b51d75d30696b742c721a2e3"
server_key_version: "encryptionkey@0"
I20260925 02:56:43.021730 21057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "ef3ac5745fdc4b1b994f479bc1c69f60"
format_stamp: "Formatted at 2026-09-25 02:56:43 on dist-test-slave-8zvk"
server_key: "cc62ecbac1e4a5974d907e082f3e8a7f"
server_key_iv: "9e9bc173b51d75d30696b742c721a2e3"
server_key_version: "encryptionkey@0"
I20260925 02:56:43.026197 21057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 02:56:43.029237 21072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:43.030469 21057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:56:43.030620 21057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "ef3ac5745fdc4b1b994f479bc1c69f60"
format_stamp: "Formatted at 2026-09-25 02:56:43 on dist-test-slave-8zvk"
server_key: "cc62ecbac1e4a5974d907e082f3e8a7f"
server_key_iv: "9e9bc173b51d75d30696b742c721a2e3"
server_key_version: "encryptionkey@0"
I20260925 02:56:43.030789 21057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:43.055018 21057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:43.058671 21057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:43.058976 21057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:43.061324 21057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:43.061450 21057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:43.061635 21057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:43.061781 21057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:56:43 dist-test-slave-8zvk krb5kdc[20941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305003, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:43.084462 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:43.061669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:33297 (local address 127.18.196.254:40507)
0925 02:56:43.061866 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:43.061873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:43.063210 (+ 1337us) server_negotiation.cc:407] Connection header received
0925 02:56:43.065185 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:43.065191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:43.065286 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:43.065392 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:43.067571 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:43.068595 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:43.071260 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:43.071947 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:43.072098 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:43.076575 (+ 4477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:43.076581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:43.076584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:43.076613 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:43.080344 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:43.081412 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:43.081421 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:43.081427 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:43.081642 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:43.082190 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:43.082198 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:43.082204 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:43.082553 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:43.082715 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:43.083635 (+ 920us) server_negotiation.cc:299] Negotiation successful
0925 02:56:43.083952 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:56:43.084928 21081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:43.062161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40507 (local address 127.18.196.193:33297)
0925 02:56:43.063040 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:43.063091 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:43.064887 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:43.065651 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:43.065666 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:43.066376 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:43.067381 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:43.067404 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:43.068812 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:43.068819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:43.071092 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:43.071113 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:43.072163 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:43.072168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:43.072355 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:43.073010 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:43.073071 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:43.076263 (+ 3192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:43.080523 (+ 4260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:43.080536 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:43.080554 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:43.081237 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:43.081762 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:43.081765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:43.081767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:43.082001 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:43.082687 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:43.082695 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:43.083111 (+ 416us) client_negotiation.cc:769] Sending connection context
0925 02:56:43.083485 (+ 374us) client_negotiation.cc:240] Negotiation successful
0925 02:56:43.083865 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":372,"threads_started":1}
I20260925 02:56:43.108994 21057 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:43197
I20260925 02:56:43.109072 21151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:43.093844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40507 (local address 127.18.196.193:58937)
0925 02:56:43.094409 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:43.094428 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:43.094571 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:43.095022 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:43.095027 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:43.095751 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:43.096285 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:43.096306 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:43.097629 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:43.097634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:43.099578 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:43.099605 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:43.100448 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:43.100455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:43.100633 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:43.101244 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:43.101271 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:43.102397 (+ 1126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:43.106273 (+ 3876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:43.106278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:43.106281 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:43.106782 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:43.107253 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:43.107258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:43.107261 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:43.107386 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:43.107861 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:43.107866 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:43.108051 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 02:56:43.108254 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 02:56:43.108562 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":324,"threads_started":1}
I20260925 02:56:43.109097 21191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:43197 every 8 connection(s)
I20260925 02:56:43.109730 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:43.093490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:58937 (local address 127.18.196.254:40507)
0925 02:56:43.093731 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:43.093740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:43.094512 (+ 772us) server_negotiation.cc:407] Connection header received
0925 02:56:43.094736 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:43.094741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:43.094878 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:43.094995 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:43.096450 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:43.097497 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:43.099796 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:43.100239 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:43.100416 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:43.102595 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:43.102603 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:43.102612 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:43.102652 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:43.106095 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:43.106964 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:43.106971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:43.106976 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:43.107140 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:43.107488 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:43.107492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:43.107496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:43.107736 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:43.107889 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:43.109359 (+ 1470us) server_negotiation.cc:299] Negotiation successful
0925 02:56:43.109514 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:56:43.110373 21057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 02:56:43.112382 21192 heartbeater.cc:347] Connected to a master server at 127.18.196.254:40507
I20260925 02:56:43.112603 21192 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:43.113185 21192 heartbeater.cc:511] Master 127.18.196.254:40507 requested a full tablet report, sending...
I20260925 02:56:43.114956 20986 ts_manager.cc:194] Registered new tserver with Master: ef3ac5745fdc4b1b994f479bc1c69f60 (127.18.196.193:43197)
I20260925 02:56:43.116149 20986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.193@KRBTEST.COM'} at 127.18.196.193:33297
I20260925 02:56:43.116734 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 21057
I20260925 02:56:43.116937 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 02:56:43.117612 19219 external_mini_cluster.cc:1428] Setting key e648c690ebce8fbd67ba54220514a055
WARNING: no policy specified for kudu/127.18.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
I20260925 02:56:43.154035 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:40507
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:43067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:43.460970 21196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:43.461202 21196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:43.461395 21196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:43.468014 21196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:43.468101 21196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:43.468261 21196 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 02:56:43.468340 21196 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:56:43.468482 21196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 02:56:43.477246 21196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:43067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.196.254:40507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:43.478313 21196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:43.479532 21196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:43.491665 21202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:43.491729 21201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:43.491727 21204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:43.492244 21196 server_base.cc:1034] running on GCE node
I20260925 02:56:43.492837 21196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:43.493721 21196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:43.494971 21196 hybrid_clock.cc:648] HybridClock initialized: now 1790305003494880 us; error 43 us; skew 500 ppm
I20260925 02:56:43.495262 21196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:43 dist-test-slave-8zvk krb5kdc[20941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305003, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:43.498049 21196 init.cc:377] Logged in from keytab as kudu/127.18.196.194@KRBTEST.COM (short username kudu)
I20260925 02:56:43.499377 21196 webserver.cc:460] Webserver started at http://127.18.196.194:33667/ using document root <none> and password file <none>
I20260925 02:56:43.499951 21196 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:43.500088 21196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:43.500389 21196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:43.503110 21196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "ea99845dd3ed4c30a08935a5123d219d"
format_stamp: "Formatted at 2026-09-25 02:56:43 on dist-test-slave-8zvk"
server_key: "e8b120ed2ee9ff7bfd1d0352299e5b37"
server_key_iv: "e8a5e23f0623dd1021c5bb66c10ddb19"
server_key_version: "encryptionkey@0"
I20260925 02:56:43.503820 21196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "ea99845dd3ed4c30a08935a5123d219d"
format_stamp: "Formatted at 2026-09-25 02:56:43 on dist-test-slave-8zvk"
server_key: "e8b120ed2ee9ff7bfd1d0352299e5b37"
server_key_iv: "e8a5e23f0623dd1021c5bb66c10ddb19"
server_key_version: "encryptionkey@0"
I20260925 02:56:43.508177 21196 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:56:43.511189 21211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:43.512277 21196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:56:43.512472 21196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "ea99845dd3ed4c30a08935a5123d219d"
format_stamp: "Formatted at 2026-09-25 02:56:43 on dist-test-slave-8zvk"
server_key: "e8b120ed2ee9ff7bfd1d0352299e5b37"
server_key_iv: "e8a5e23f0623dd1021c5bb66c10ddb19"
server_key_version: "encryptionkey@0"
I20260925 02:56:43.512696 21196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:43.535048 21196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:43.538311 21196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:43.538517 21196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:43.540917 21196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:43.541047 21196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:43.541184 21196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:43.541298 21196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:56:43 dist-test-slave-8zvk krb5kdc[20941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305003, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:43.566099 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:43.541664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:45665 (local address 127.18.196.254:40507)
0925 02:56:43.541825 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:43.541831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:43.543452 (+ 1621us) server_negotiation.cc:407] Connection header received
0925 02:56:43.546014 (+ 2562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:43.546020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:43.546092 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:43.546150 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:43.548315 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:43.549469 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:43.551927 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:43.552577 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:43.552670 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:43.557848 (+ 5178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:43.557855 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:43.557859 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:43.557894 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:43.561212 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:43.562168 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:43.562176 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:43.562182 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:43.562337 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:43.562940 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:43.562948 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:43.562951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:43.563369 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:43.563539 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:43.565417 (+ 1878us) server_negotiation.cc:299] Negotiation successful
0925 02:56:43.565696 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:56:43.566501 21221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:43.542349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40507 (local address 127.18.196.194:45665)
0925 02:56:43.543275 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:43.543326 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:43.545701 (+ 2375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:43.546616 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:43.546626 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:43.547307 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:43.548080 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:43.548101 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:43.549700 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:43.549705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:43.551761 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:43.551780 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:43.552975 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:43.552985 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:43.553331 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:43.554060 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:43.554095 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:43.557546 (+ 3451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:43.561376 (+ 3830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:43.561393 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:43.561412 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:43.561963 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:43.562467 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:43.562472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:43.562478 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:43.562762 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:43.563539 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:43.563548 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:43.563875 (+ 327us) client_negotiation.cc:769] Sending connection context
0925 02:56:43.564229 (+ 354us) client_negotiation.cc:240] Negotiation successful
0925 02:56:43.564689 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":423,"threads_started":1}
I20260925 02:56:43.587263 21196 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:45877
I20260925 02:56:43.587353 21332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:45877 every 8 connection(s)
I20260925 02:56:43.588042 21297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:43.573211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40507 (local address 127.18.196.194:35441)
0925 02:56:43.573803 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:43.573838 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:43.573982 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:43.574336 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:43.574339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:43.574713 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:43.575138 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:43.575152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:43.576426 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:43.576432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:43.577900 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:43.577912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:43.578767 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:43.578778 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:43.579121 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:43.579769 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:43.579794 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:43.581499 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:43.585000 (+ 3501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:43.585004 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:43.585008 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:43.585579 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:43.586051 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:43.586055 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:43.586057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:43.586224 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:43.587087 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:43.587090 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:43.587353 (+ 263us) client_negotiation.cc:769] Sending connection context
0925 02:56:43.587498 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 02:56:43.587744 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":558,"threads_started":1}
I20260925 02:56:43.588490 21196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 02:56:43.588883 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:43.573593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:35441 (local address 127.18.196.254:40507)
0925 02:56:43.573756 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:43.573761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:43.573938 (+ 177us) server_negotiation.cc:407] Connection header received
0925 02:56:43.574120 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:43.574128 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:43.574199 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:43.574301 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:43.575338 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:43.576260 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:43.578089 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:43.578543 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:43.578678 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:43.581700 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:43.581709 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:43.581715 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:43.581758 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:43.584873 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:43.585683 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:43.585705 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:43.585709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:43.585864 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:43.586402 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:43.586408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:43.586413 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:43.586862 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:43.587021 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:43.588498 (+ 1477us) server_negotiation.cc:299] Negotiation successful
0925 02:56:43.588694 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:56:43.590348 21333 heartbeater.cc:347] Connected to a master server at 127.18.196.254:40507
I20260925 02:56:43.590557 21333 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:43.591220 21333 heartbeater.cc:511] Master 127.18.196.254:40507 requested a full tablet report, sending...
I20260925 02:56:43.592537 20986 ts_manager.cc:194] Registered new tserver with Master: ea99845dd3ed4c30a08935a5123d219d (127.18.196.194:45877)
I20260925 02:56:43.593291 20986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.194@KRBTEST.COM'} at 127.18.196.194:45665
I20260925 02:56:43.598814 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 21196
I20260925 02:56:43.599021 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 02:56:43.599488 19219 external_mini_cluster.cc:1428] Setting key c29b0ac704c3d551d737297803b4711d
WARNING: no policy specified for kudu/127.18.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
I20260925 02:56:43.637979 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:40507
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:43067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:43.942348 21337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:43.942606 21337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:43.942765 21337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:43.949626 21337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:43.949716 21337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:43.949828 21337 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 02:56:43.949944 21337 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:56:43.950084 21337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 02:56:43.959481 21337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:43067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.196.254:40507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:43.960669 21337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:43.962036 21337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:43.975051 21342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:43.975143 21345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:43.975099 21343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:43.975981 21337 server_base.cc:1034] running on GCE node
I20260925 02:56:43.976555 21337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:43.977437 21337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:43.978663 21337 hybrid_clock.cc:648] HybridClock initialized: now 1790305003978630 us; error 48 us; skew 500 ppm
I20260925 02:56:43.979032 21337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:43 dist-test-slave-8zvk krb5kdc[20941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305003, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:43.985721 21337 init.cc:377] Logged in from keytab as kudu/127.18.196.195@KRBTEST.COM (short username kudu)
I20260925 02:56:43.987329 21337 webserver.cc:460] Webserver started at http://127.18.196.195:43879/ using document root <none> and password file <none>
I20260925 02:56:43.987977 21337 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:43.988118 21337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:43.988448 21337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:43.991308 21337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "d1bf63069d8f4d0da52508f00ea11746"
format_stamp: "Formatted at 2026-09-25 02:56:43 on dist-test-slave-8zvk"
server_key: "392a81555a6f4b99ddf163ebc2540b90"
server_key_iv: "8eeba84db949147e6e286b73f4948b44"
server_key_version: "encryptionkey@0"
I20260925 02:56:43.992141 21337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "d1bf63069d8f4d0da52508f00ea11746"
format_stamp: "Formatted at 2026-09-25 02:56:43 on dist-test-slave-8zvk"
server_key: "392a81555a6f4b99ddf163ebc2540b90"
server_key_iv: "8eeba84db949147e6e286b73f4948b44"
server_key_version: "encryptionkey@0"
I20260925 02:56:43.997328 21337 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 02:56:44.000681 21352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:44.001979 21337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:56:44.002147 21337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "d1bf63069d8f4d0da52508f00ea11746"
format_stamp: "Formatted at 2026-09-25 02:56:43 on dist-test-slave-8zvk"
server_key: "392a81555a6f4b99ddf163ebc2540b90"
server_key_iv: "8eeba84db949147e6e286b73f4948b44"
server_key_version: "encryptionkey@0"
I20260925 02:56:44.002357 21337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:44.019431 21337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:44.023258 21337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:44.023540 21337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:44.025979 21337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:44.026104 21337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:44.026217 21337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:44.026305 21337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:56:44 dist-test-slave-8zvk krb5kdc[20941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305003, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:44.049517 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.026604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:42719 (local address 127.18.196.254:40507)
0925 02:56:44.026810 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:44.026816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:44.028221 (+ 1405us) server_negotiation.cc:407] Connection header received
0925 02:56:44.030408 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:44.030415 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:44.030513 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:44.030650 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:44.032961 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.033916 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.036617 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.037033 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.037182 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.041876 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:44.041885 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:44.041891 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:44.041936 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:44.046019 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:44.047146 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:44.047150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:44.047152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:44.047271 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:44.047835 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:44.047839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:44.047843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:44.048200 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:44.048349 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:44.048986 (+ 637us) server_negotiation.cc:299] Negotiation successful
0925 02:56:44.049191 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 02:56:44.050604 21361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.027300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40507 (local address 127.18.196.195:42719)
0925 02:56:44.028068 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:44.028108 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:44.030087 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:44.031113 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:44.031122 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:44.031860 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:44.032780 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.032810 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.034124 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.034130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.036445 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.036466 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.037312 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.037322 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.037624 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.038406 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:44.038442 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:44.041578 (+ 3136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:44.046282 (+ 4704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:44.046294 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:44.046314 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:44.046985 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:44.047432 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:44.047439 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:44.047443 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:44.047710 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:44.048326 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:44.048336 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:44.048652 (+ 316us) client_negotiation.cc:769] Sending connection context
0925 02:56:44.048879 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 02:56:44.049336 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":331,"threads_started":1}
I20260925 02:56:44.069945 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.056948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:50067 (local address 127.18.196.254:40507)
0925 02:56:44.057104 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:44.057108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:44.057976 (+ 868us) server_negotiation.cc:407] Connection header received
0925 02:56:44.058077 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:44.058084 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:44.058178 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:44.058294 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:44.059242 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.060253 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.062106 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.062557 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.062696 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.064600 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:44.064606 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:44.064610 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:44.064653 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:44.067095 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:44.067780 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:44.067786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:44.067790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:44.067956 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:44.068484 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:44.068488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:44.068490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:44.068748 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:44.068901 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:44.069345 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 02:56:44.069590 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:56:44.069976 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.057093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40507 (local address 127.18.196.195:50067)
0925 02:56:44.057878 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:44.057892 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:44.057989 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:44.058306 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:44.058309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:44.058719 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:44.059093 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.059107 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.060390 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.060400 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.061949 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.061964 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.062773 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.062781 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.062980 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.063610 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:44.063626 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:44.064428 (+ 802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:44.067224 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:44.067229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:44.067233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:44.067653 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:44.068090 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:44.068102 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:44.068108 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:44.068332 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:44.068870 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:44.068875 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:44.069057 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 02:56:44.069203 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 02:56:44.069565 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":510,"threads_started":1}
I20260925 02:56:44.077387 21337 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:44077
I20260925 02:56:44.077459 21472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:44077 every 8 connection(s)
I20260925 02:56:44.078488 21337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 02:56:44.080435 21473 heartbeater.cc:347] Connected to a master server at 127.18.196.254:40507
I20260925 02:56:44.080703 21473 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:44.081339 21473 heartbeater.cc:511] Master 127.18.196.254:40507 requested a full tablet report, sending...
I20260925 02:56:44.082576 20986 ts_manager.cc:194] Registered new tserver with Master: d1bf63069d8f4d0da52508f00ea11746 (127.18.196.195:44077)
I20260925 02:56:44.083360 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 21337
I20260925 02:56:44.083545 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 02:56:44.083639 20986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.195@KRBTEST.COM'} at 127.18.196.195:42719
I20260925 02:56:44.084125 19219 external_mini_cluster.cc:1428] Setting key 1300ab7f704561b3f7db49c1e87e21ba
I20260925 02:56:44.087009 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:56:44 dist-test-slave-8zvk krb5kdc[20941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260925 02:56:44.118808 21192 heartbeater.cc:503] Master 127.18.196.254:40507 was elected leader, sending a full tablet report...
Sep 25 02:56:44 dist-test-slave-8zvk krb5kdc[20941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:44.133737 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.120612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43672 (local address 127.18.196.254:40507)
0925 02:56:44.120865 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:44.120871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:44.121173 (+ 302us) server_negotiation.cc:407] Connection header received
0925 02:56:44.121391 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:44.121396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:44.121472 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:44.121579 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:44.123086 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.123792 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.124981 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.125423 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.125561 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.129045 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:44.129055 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:44.129058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:44.129085 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:44.131215 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:44.131812 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:44.131818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:44.131822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:44.131951 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:44.132342 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:44.132347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:44.132350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:44.132618 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:44.132879 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:44.133178 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 02:56:44.133391 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260925 02:56:44.137902 20986 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:56:44.144675 20986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:56:44.191385 21486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.156212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:44077 (local address 127.0.0.1:56066)
0925 02:56:44.182430 (+ 26218us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:44.182450 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:44.182567 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:44.183940 (+ 1373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:44.183944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:44.183967 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:44.184375 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.184387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.186258 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.186263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.188689 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.188698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.190337 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.190343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.190536 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.190590 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 02:56:44.190739 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 02:56:44.190968 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26096,"mutex_wait_us":55,"thread_start_us":27481,"threads_started":1}
I20260925 02:56:44.192201 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.156715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56066 (local address 127.18.196.195:44077)
0925 02:56:44.181884 (+ 25169us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:44.181892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:44.182943 (+ 1051us) server_negotiation.cc:407] Connection header received
0925 02:56:44.183043 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:44.183053 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:44.183272 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:44.183441 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:44.184588 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.186111 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.188839 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.190118 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.190335 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.190547 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:44.191678 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0925 02:56:44.191864 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24999,"thread_start_us":25225,"threads_started":1}
I20260925 02:56:44.192418 21487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.156686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:43197 (local address 127.0.0.1:60902)
0925 02:56:44.184957 (+ 28271us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:44.184977 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:44.185113 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:44.185659 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:44.185663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:44.185687 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:44.186067 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.186077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.187837 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.187841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.189533 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.189548 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.191307 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.191315 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.191503 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.191792 (+ 289us) client_negotiation.cc:769] Sending connection context
0925 02:56:44.191902 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 02:56:44.192047 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28148,"thread_start_us":25825,"threads_started":1}
I20260925 02:56:44.192862 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.157305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60902 (local address 127.18.196.193:43197)
0925 02:56:44.180607 (+ 23302us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:44.180620 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:44.185093 (+ 4473us) server_negotiation.cc:407] Connection header received
0925 02:56:44.185279 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:44.185284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:44.185484 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:44.185669 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:44.186235 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.187682 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.189687 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.190811 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.191004 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.191161 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:44.191978 (+ 817us) server_negotiation.cc:299] Negotiation successful
0925 02:56:44.192200 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23055,"thread_start_us":23310,"threads_started":1}
I20260925 02:56:44.193178 21489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.156156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:45877 (local address 127.0.0.1:59996)
0925 02:56:44.181533 (+ 25377us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:44.181587 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:44.181804 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:44.183455 (+ 1651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:44.183463 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:44.183505 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:44.183949 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.183961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.185878 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.185883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.188670 (+ 2787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.188690 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.192348 (+ 3658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.192356 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.192558 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.192591 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 02:56:44.192688 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 02:56:44.192813 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25044,"mutex_wait_us":81,"thread_start_us":28537,"threads_started":1}
I20260925 02:56:44.193970 21488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.156266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59996 (local address 127.18.196.194:45877)
0925 02:56:44.182542 (+ 26276us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:44.182551 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:44.182581 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:56:44.182686 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:44.182694 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:44.183008 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:44.183193 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:44.184156 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.185723 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.190629 (+ 4906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.191802 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.192016 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.192177 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:44.193563 (+ 1386us) server_negotiation.cc:299] Negotiation successful
0925 02:56:44.193729 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26098,"thread_start_us":25994,"threads_started":1}
I20260925 02:56:44.196233 21122 tablet_service.cc:1467] Processing CreateTablet for tablet 9b758ea106b14f9db786941b8b10929c (DEFAULT_TABLE table=test-table [id=6d2d641387fe4eb596d4dcff1fc259cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:44.196305 21401 tablet_service.cc:1467] Processing CreateTablet for tablet 9b758ea106b14f9db786941b8b10929c (DEFAULT_TABLE table=test-table [id=6d2d641387fe4eb596d4dcff1fc259cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:44.197109 21262 tablet_service.cc:1467] Processing CreateTablet for tablet 9b758ea106b14f9db786941b8b10929c (DEFAULT_TABLE table=test-table [id=6d2d641387fe4eb596d4dcff1fc259cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:44.197587 21122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b758ea106b14f9db786941b8b10929c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:44.197692 21401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b758ea106b14f9db786941b8b10929c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:44.198253 21262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b758ea106b14f9db786941b8b10929c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:44.209060 21492 tablet_bootstrap.cc:492] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60: Bootstrap starting.
I20260925 02:56:44.211794 21493 tablet_bootstrap.cc:492] T 9b758ea106b14f9db786941b8b10929c P d1bf63069d8f4d0da52508f00ea11746: Bootstrap starting.
I20260925 02:56:44.213011 21492 tablet_bootstrap.cc:654] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:44.214351 21492 log.cc:824] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:44.216094 21493 tablet_bootstrap.cc:654] T 9b758ea106b14f9db786941b8b10929c P d1bf63069d8f4d0da52508f00ea11746: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:44.216347 21494 tablet_bootstrap.cc:492] T 9b758ea106b14f9db786941b8b10929c P ea99845dd3ed4c30a08935a5123d219d: Bootstrap starting.
I20260925 02:56:44.217374 21493 log.cc:824] T 9b758ea106b14f9db786941b8b10929c P d1bf63069d8f4d0da52508f00ea11746: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:44.217491 21492 tablet_bootstrap.cc:492] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60: No bootstrap required, opened a new log
I20260925 02:56:44.217739 21492 ts_tablet_manager.cc:1397] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260925 02:56:44.219306 21494 tablet_bootstrap.cc:654] T 9b758ea106b14f9db786941b8b10929c P ea99845dd3ed4c30a08935a5123d219d: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:44.220170 21494 log.cc:824] T 9b758ea106b14f9db786941b8b10929c P ea99845dd3ed4c30a08935a5123d219d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:44.223277 21494 tablet_bootstrap.cc:492] T 9b758ea106b14f9db786941b8b10929c P ea99845dd3ed4c30a08935a5123d219d: No bootstrap required, opened a new log
I20260925 02:56:44.223276 21493 tablet_bootstrap.cc:492] T 9b758ea106b14f9db786941b8b10929c P d1bf63069d8f4d0da52508f00ea11746: No bootstrap required, opened a new log
I20260925 02:56:44.223546 21493 ts_tablet_manager.cc:1397] T 9b758ea106b14f9db786941b8b10929c P d1bf63069d8f4d0da52508f00ea11746: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260925 02:56:44.223546 21494 ts_tablet_manager.cc:1397] T 9b758ea106b14f9db786941b8b10929c P ea99845dd3ed4c30a08935a5123d219d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 02:56:44.226365 21492 raft_consensus.cc:348] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } } peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } }
I20260925 02:56:44.226699 21492 raft_consensus.cc:374] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:44.226820 21492 raft_consensus.cc:729] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3ac5745fdc4b1b994f479bc1c69f60, State: Initialized, Role: FOLLOWER
I20260925 02:56:44.227411 21492 consensus_queue.cc:260] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } } peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } }
I20260925 02:56:44.229017 21492 ts_tablet_manager.cc:1428] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260925 02:56:44.233803 21494 raft_consensus.cc:348] T 9b758ea106b14f9db786941b8b10929c P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } } peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } }
I20260925 02:56:44.233803 21493 raft_consensus.cc:348] T 9b758ea106b14f9db786941b8b10929c P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } } peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } }
I20260925 02:56:44.234208 21493 raft_consensus.cc:374] T 9b758ea106b14f9db786941b8b10929c P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:44.234208 21494 raft_consensus.cc:374] T 9b758ea106b14f9db786941b8b10929c P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:44.234309 21493 raft_consensus.cc:729] T 9b758ea106b14f9db786941b8b10929c P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1bf63069d8f4d0da52508f00ea11746, State: Initialized, Role: FOLLOWER
I20260925 02:56:44.234309 21494 raft_consensus.cc:729] T 9b758ea106b14f9db786941b8b10929c P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea99845dd3ed4c30a08935a5123d219d, State: Initialized, Role: FOLLOWER
I20260925 02:56:44.234910 21493 consensus_queue.cc:260] T 9b758ea106b14f9db786941b8b10929c P d1bf63069d8f4d0da52508f00ea11746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } } peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } }
I20260925 02:56:44.234898 21494 consensus_queue.cc:260] T 9b758ea106b14f9db786941b8b10929c P ea99845dd3ed4c30a08935a5123d219d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } } peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } }
I20260925 02:56:44.235888 21333 heartbeater.cc:503] Master 127.18.196.254:40507 was elected leader, sending a full tablet report...
I20260925 02:56:44.236306 21473 heartbeater.cc:503] Master 127.18.196.254:40507 was elected leader, sending a full tablet report...
I20260925 02:56:44.236598 21494 ts_tablet_manager.cc:1428] T 9b758ea106b14f9db786941b8b10929c P ea99845dd3ed4c30a08935a5123d219d: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260925 02:56:44.236873 21493 ts_tablet_manager.cc:1428] T 9b758ea106b14f9db786941b8b10929c P d1bf63069d8f4d0da52508f00ea11746: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 02:56:44.247184 21498 raft_consensus.cc:482] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:56:44.247386 21498 raft_consensus.cc:504] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } } peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } }
I20260925 02:56:44.248829 21498 leader_election.cc:288] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea99845dd3ed4c30a08935a5123d219d (127.18.196.194:45877), d1bf63069d8f4d0da52508f00ea11746 (127.18.196.195:44077)
I20260925 02:56:44.255899 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.249316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:43413 (local address 127.18.196.195:44077)
0925 02:56:44.249488 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:44.249494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:44.250051 (+ 557us) server_negotiation.cc:407] Connection header received
0925 02:56:44.250216 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:44.250219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:44.250292 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:44.250376 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:44.251234 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.251919 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.254047 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.254903 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.255050 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.255136 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:44.255500 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 02:56:44.255630 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 02:56:44.255942 21488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.249046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:57109 (local address 127.18.196.194:45877)
0925 02:56:44.249263 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:44.249272 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:44.249972 (+ 700us) server_negotiation.cc:407] Connection header received
0925 02:56:44.250144 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:44.250150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:44.250227 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:44.250286 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:44.251192 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.251896 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.254120 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.254973 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.255135 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.255206 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:44.255551 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 02:56:44.255668 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:56:44.256215 21501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.249117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:44077 (local address 127.18.196.193:43413)
0925 02:56:44.249916 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:44.249953 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:44.250099 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:44.250409 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:44.250413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:44.250443 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:44.251054 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.251066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.252101 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.252105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.253874 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.253889 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.255120 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.255130 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.255309 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.255369 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 02:56:44.255444 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 02:56:44.255593 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":77,"thread_start_us":415,"threads_started":1}
I20260925 02:56:44.256271 21502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.248963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:45877 (local address 127.18.196.193:57109)
0925 02:56:44.249867 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:44.249881 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:44.249994 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:44.250328 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:44.250335 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:44.250364 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:44.251053 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.251066 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.252034 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.252039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.253935 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.253976 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.255201 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.255206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.255372 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.255395 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:56:44.255504 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 02:56:44.255629 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":42,"spinlock_wait_cycles":16640,"thread_start_us":660,"threads_started":1}
I20260925 02:56:44.257094 21282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b758ea106b14f9db786941b8b10929c" candidate_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea99845dd3ed4c30a08935a5123d219d" is_pre_election: true
I20260925 02:56:44.257097 21422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b758ea106b14f9db786941b8b10929c" candidate_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1bf63069d8f4d0da52508f00ea11746" is_pre_election: true
I20260925 02:56:44.257522 21422 raft_consensus.cc:2454] T 9b758ea106b14f9db786941b8b10929c P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef3ac5745fdc4b1b994f479bc1c69f60 in term 0.
I20260925 02:56:44.257522 21282 raft_consensus.cc:2454] T 9b758ea106b14f9db786941b8b10929c P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef3ac5745fdc4b1b994f479bc1c69f60 in term 0.
I20260925 02:56:44.258121 21074 leader_election.cc:302] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: d1bf63069d8f4d0da52508f00ea11746, ea99845dd3ed4c30a08935a5123d219d, ef3ac5745fdc4b1b994f479bc1c69f60; no voters:
I20260925 02:56:44.258473 21498 raft_consensus.cc:2781] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:56:44.258621 21498 raft_consensus.cc:482] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:56:44.258734 21498 raft_consensus.cc:3037] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:44.261415 21498 raft_consensus.cc:504] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } } peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } }
I20260925 02:56:44.262044 21498 leader_election.cc:288] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60 [CANDIDATE]: Term 1 election: Requested vote from peers ea99845dd3ed4c30a08935a5123d219d (127.18.196.194:45877), d1bf63069d8f4d0da52508f00ea11746 (127.18.196.195:44077)
I20260925 02:56:44.262532 21282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b758ea106b14f9db786941b8b10929c" candidate_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea99845dd3ed4c30a08935a5123d219d"
I20260925 02:56:44.262539 21422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b758ea106b14f9db786941b8b10929c" candidate_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1bf63069d8f4d0da52508f00ea11746"
I20260925 02:56:44.262739 21422 raft_consensus.cc:3037] T 9b758ea106b14f9db786941b8b10929c P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:44.262739 21282 raft_consensus.cc:3037] T 9b758ea106b14f9db786941b8b10929c P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:44.265969 21422 raft_consensus.cc:2454] T 9b758ea106b14f9db786941b8b10929c P d1bf63069d8f4d0da52508f00ea11746 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef3ac5745fdc4b1b994f479bc1c69f60 in term 1.
I20260925 02:56:44.265969 21282 raft_consensus.cc:2454] T 9b758ea106b14f9db786941b8b10929c P ea99845dd3ed4c30a08935a5123d219d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef3ac5745fdc4b1b994f479bc1c69f60 in term 1.
I20260925 02:56:44.266455 21074 leader_election.cc:302] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1bf63069d8f4d0da52508f00ea11746, ef3ac5745fdc4b1b994f479bc1c69f60; no voters:
I20260925 02:56:44.266745 21498 raft_consensus.cc:2781] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:44.267198 21498 raft_consensus.cc:686] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60 [term 1 LEADER]: Becoming Leader. State: Replica: ef3ac5745fdc4b1b994f479bc1c69f60, State: Running, Role: LEADER
I20260925 02:56:44.267920 21498 consensus_queue.cc:237] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } } peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } }
I20260925 02:56:44.272714 20986 catalog_manager.cc:5495] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef3ac5745fdc4b1b994f479bc1c69f60 (127.18.196.193). New cstate: current_term: 1 leader_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } health_report { overall_health: HEALTHY } } }
Sep 25 02:56:44 dist-test-slave-8zvk krb5kdc[20941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305002, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:44.318312 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.305144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43686 (local address 127.18.196.254:40507)
0925 02:56:44.305345 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:44.305349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:44.305869 (+ 520us) server_negotiation.cc:407] Connection header received
0925 02:56:44.306073 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:44.306078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:44.306166 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:44.306251 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:44.307438 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.308158 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.309358 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.309760 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.309894 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.312980 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:44.312985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:44.312989 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:44.313015 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:44.315274 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:44.316178 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:44.316185 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:44.316188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:44.316339 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:44.316701 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:44.316706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:44.316709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:44.316961 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:44.317116 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:44.317658 (+ 542us) server_negotiation.cc:299] Negotiation successful
0925 02:56:44.317981 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":49}
I20260925 02:56:44.318348 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.305195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40507 (local address 127.0.0.1:43686)
0925 02:56:44.305752 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:44.305767 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:44.305946 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:44.306274 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:44.306279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:44.307020 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:44.307291 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.307300 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.308282 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.308288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.309215 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.309226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.309912 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.309918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.310112 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.310688 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:44.310705 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:44.312798 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:44.315392 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:44.315405 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:44.315413 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:44.315994 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:44.316444 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:44.316450 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:44.316453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:44.316589 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:44.317110 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:44.317118 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:44.317394 (+ 276us) client_negotiation.cc:769] Sending connection context
0925 02:56:44.317564 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 02:56:44.317857 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":316,"threads_started":1}
I20260925 02:56:44.323196 20986 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:43686:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 02:56:44.324084 20986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:56:44.330314 21122 tablet_service.cc:1467] Processing CreateTablet for tablet edadedc44800410296124641774429fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2096cceeb6824b58b98423123cd53764]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:56:44.330540 21262 tablet_service.cc:1467] Processing CreateTablet for tablet edadedc44800410296124641774429fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2096cceeb6824b58b98423123cd53764]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:56:44.330540 21401 tablet_service.cc:1467] Processing CreateTablet for tablet edadedc44800410296124641774429fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2096cceeb6824b58b98423123cd53764]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:56:44.331228 21122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edadedc44800410296124641774429fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:44.331442 21401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edadedc44800410296124641774429fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:44.331456 21262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edadedc44800410296124641774429fa. 1 dirs total, 0 dirs full, 0 dirs failed
W20260925 02:56:44.333688 21474 tablet.cc:2369] T 9b758ea106b14f9db786941b8b10929c P d1bf63069d8f4d0da52508f00ea11746: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:56:44.336901 21492 tablet_bootstrap.cc:492] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60: Bootstrap starting.
I20260925 02:56:44.338198 21492 tablet_bootstrap.cc:654] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:44.338493 21493 tablet_bootstrap.cc:492] T edadedc44800410296124641774429fa P d1bf63069d8f4d0da52508f00ea11746: Bootstrap starting.
I20260925 02:56:44.338570 21494 tablet_bootstrap.cc:492] T edadedc44800410296124641774429fa P ea99845dd3ed4c30a08935a5123d219d: Bootstrap starting.
I20260925 02:56:44.340111 21493 tablet_bootstrap.cc:654] T edadedc44800410296124641774429fa P d1bf63069d8f4d0da52508f00ea11746: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:44.340695 21494 tablet_bootstrap.cc:654] T edadedc44800410296124641774429fa P ea99845dd3ed4c30a08935a5123d219d: Neither blocks nor log segments found. Creating new log.
W20260925 02:56:44.341229 21334 tablet.cc:2369] T 9b758ea106b14f9db786941b8b10929c P ea99845dd3ed4c30a08935a5123d219d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:56:44.344916 21492 tablet_bootstrap.cc:492] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60: No bootstrap required, opened a new log
I20260925 02:56:44.344928 21493 tablet_bootstrap.cc:492] T edadedc44800410296124641774429fa P d1bf63069d8f4d0da52508f00ea11746: No bootstrap required, opened a new log
I20260925 02:56:44.345074 21492 ts_tablet_manager.cc:1397] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 02:56:44.345106 21493 ts_tablet_manager.cc:1397] T edadedc44800410296124641774429fa P d1bf63069d8f4d0da52508f00ea11746: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260925 02:56:44.345307 21494 tablet_bootstrap.cc:492] T edadedc44800410296124641774429fa P ea99845dd3ed4c30a08935a5123d219d: No bootstrap required, opened a new log
I20260925 02:56:44.345434 21494 ts_tablet_manager.cc:1397] T edadedc44800410296124641774429fa P ea99845dd3ed4c30a08935a5123d219d: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260925 02:56:44.345810 21493 raft_consensus.cc:348] T edadedc44800410296124641774429fa P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:44.345865 21492 raft_consensus.cc:348] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:44.346026 21493 raft_consensus.cc:374] T edadedc44800410296124641774429fa P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:44.346089 21492 raft_consensus.cc:374] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:44.346118 21493 raft_consensus.cc:729] T edadedc44800410296124641774429fa P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1bf63069d8f4d0da52508f00ea11746, State: Initialized, Role: FOLLOWER
I20260925 02:56:44.346180 21492 raft_consensus.cc:729] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3ac5745fdc4b1b994f479bc1c69f60, State: Initialized, Role: FOLLOWER
I20260925 02:56:44.346129 21494 raft_consensus.cc:348] T edadedc44800410296124641774429fa P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:44.346335 21494 raft_consensus.cc:374] T edadedc44800410296124641774429fa P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:44.346477 21494 raft_consensus.cc:729] T edadedc44800410296124641774429fa P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea99845dd3ed4c30a08935a5123d219d, State: Initialized, Role: FOLLOWER
I20260925 02:56:44.346328 21493 consensus_queue.cc:260] T edadedc44800410296124641774429fa P d1bf63069d8f4d0da52508f00ea11746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:44.346410 21492 consensus_queue.cc:260] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:44.346771 21494 consensus_queue.cc:260] T edadedc44800410296124641774429fa P ea99845dd3ed4c30a08935a5123d219d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:44.346990 21493 ts_tablet_manager.cc:1428] T edadedc44800410296124641774429fa P d1bf63069d8f4d0da52508f00ea11746: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260925 02:56:44.347012 21492 ts_tablet_manager.cc:1428] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:56:44.346866 21499 tablet_replica.cc:439] T edadedc44800410296124641774429fa P d1bf63069d8f4d0da52508f00ea11746: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } } }
I20260925 02:56:44.347208 21499 tablet_replica.cc:442] T edadedc44800410296124641774429fa P d1bf63069d8f4d0da52508f00ea11746: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:56:44.347355 21494 ts_tablet_manager.cc:1428] T edadedc44800410296124641774429fa P ea99845dd3ed4c30a08935a5123d219d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:56:44.347270 21498 tablet_replica.cc:439] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } } }
I20260925 02:56:44.347539 21498 tablet_replica.cc:442] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:56:44.348014 21500 tablet_replica.cc:439] T edadedc44800410296124641774429fa P ea99845dd3ed4c30a08935a5123d219d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } } }
I20260925 02:56:44.348331 21500 tablet_replica.cc:442] T edadedc44800410296124641774429fa P ea99845dd3ed4c30a08935a5123d219d: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 02:56:44.362553 21193 tablet.cc:2369] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:56:44.621779 21498 raft_consensus.cc:482] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:56:44.622020 21498 raft_consensus.cc:504] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:44.622689 21498 leader_election.cc:288] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea99845dd3ed4c30a08935a5123d219d (127.18.196.194:45877), d1bf63069d8f4d0da52508f00ea11746 (127.18.196.195:44077)
I20260925 02:56:44.623297 21282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edadedc44800410296124641774429fa" candidate_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea99845dd3ed4c30a08935a5123d219d" is_pre_election: true
I20260925 02:56:44.623302 21422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edadedc44800410296124641774429fa" candidate_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1bf63069d8f4d0da52508f00ea11746" is_pre_election: true
I20260925 02:56:44.623546 21422 raft_consensus.cc:2454] T edadedc44800410296124641774429fa P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef3ac5745fdc4b1b994f479bc1c69f60 in term 0.
I20260925 02:56:44.623560 21282 raft_consensus.cc:2454] T edadedc44800410296124641774429fa P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef3ac5745fdc4b1b994f479bc1c69f60 in term 0.
I20260925 02:56:44.623992 21074 leader_election.cc:302] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1bf63069d8f4d0da52508f00ea11746, ef3ac5745fdc4b1b994f479bc1c69f60; no voters:
I20260925 02:56:44.624274 21498 raft_consensus.cc:2781] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:56:44.624359 21498 raft_consensus.cc:482] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:56:44.624430 21498 raft_consensus.cc:3037] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:44.626377 21498 raft_consensus.cc:504] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:44.626963 21498 leader_election.cc:288] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60 [CANDIDATE]: Term 1 election: Requested vote from peers ea99845dd3ed4c30a08935a5123d219d (127.18.196.194:45877), d1bf63069d8f4d0da52508f00ea11746 (127.18.196.195:44077)
I20260925 02:56:44.627413 21282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edadedc44800410296124641774429fa" candidate_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea99845dd3ed4c30a08935a5123d219d"
I20260925 02:56:44.627442 21422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edadedc44800410296124641774429fa" candidate_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1bf63069d8f4d0da52508f00ea11746"
I20260925 02:56:44.627629 21282 raft_consensus.cc:3037] T edadedc44800410296124641774429fa P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:44.627660 21422 raft_consensus.cc:3037] T edadedc44800410296124641774429fa P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:44.629733 21422 raft_consensus.cc:2454] T edadedc44800410296124641774429fa P d1bf63069d8f4d0da52508f00ea11746 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef3ac5745fdc4b1b994f479bc1c69f60 in term 1.
I20260925 02:56:44.629920 21282 raft_consensus.cc:2454] T edadedc44800410296124641774429fa P ea99845dd3ed4c30a08935a5123d219d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef3ac5745fdc4b1b994f479bc1c69f60 in term 1.
I20260925 02:56:44.630146 21074 leader_election.cc:302] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1bf63069d8f4d0da52508f00ea11746, ef3ac5745fdc4b1b994f479bc1c69f60; no voters:
I20260925 02:56:44.630438 21498 raft_consensus.cc:2781] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:44.630625 21498 raft_consensus.cc:686] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60 [term 1 LEADER]: Becoming Leader. State: Replica: ef3ac5745fdc4b1b994f479bc1c69f60, State: Running, Role: LEADER
I20260925 02:56:44.630903 21498 consensus_queue.cc:237] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:44.632434 21503 tablet_replica.cc:439] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60: TxnStatusTablet state changed. Reason: New leader ef3ac5745fdc4b1b994f479bc1c69f60. Latest consensus state: current_term: 1 leader_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } } }
I20260925 02:56:44.632752 21503 tablet_replica.cc:442] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:56:44.634081 20984 catalog_manager.cc:5495] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef3ac5745fdc4b1b994f479bc1c69f60 (127.18.196.193). New cstate: current_term: 1 leader_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } health_report { overall_health: UNKNOWN } } }
I20260925 02:56:44.647089 21515 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 02:56:44.647432 21515 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 02:56:44.695252 21503 consensus_queue.cc:1035] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:44.704272 21498 consensus_queue.cc:1035] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:44.716440 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.709533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:43197 (local address 127.0.0.1:60908)
0925 02:56:44.709734 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:44.709755 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:44.709919 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:44.710771 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:44.710776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:44.710789 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:56:44.711031 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.711044 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.712396 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.712402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.713395 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.713407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.714163 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.714169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.714331 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.714381 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:56:44.715825 (+ 1444us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:56:44.715850 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 02:56:44.715924 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 02:56:44.716114 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 02:56:44.717408 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.709673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60908 (local address 127.18.196.193:43197)
0925 02:56:44.710302 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:44.710307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:44.710328 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:56:44.710408 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:44.710415 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:44.710539 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:44.711180 (+ 641us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:44.711429 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.712230 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.713564 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.713995 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.714145 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.714576 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:44.715375 (+ 799us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:44.716857 (+ 1482us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:44.716914 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 02:56:44.716996 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":408,"threads_started":1}
I20260925 02:56:44.724069 21422 raft_consensus.cc:1261] T edadedc44800410296124641774429fa P d1bf63069d8f4d0da52508f00ea11746 [term 1 FOLLOWER]: Refusing update from remote peer ef3ac5745fdc4b1b994f479bc1c69f60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:44.724243 21282 raft_consensus.cc:1261] T edadedc44800410296124641774429fa P ea99845dd3ed4c30a08935a5123d219d [term 1 FOLLOWER]: Refusing update from remote peer ef3ac5745fdc4b1b994f479bc1c69f60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:44.724661 21519 consensus_queue.cc:1035] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:44.725061 21503 consensus_queue.cc:1035] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:44.739614 21499 tablet_replica.cc:439] T edadedc44800410296124641774429fa P d1bf63069d8f4d0da52508f00ea11746: TxnStatusTablet state changed. Reason: New leader ef3ac5745fdc4b1b994f479bc1c69f60. Latest consensus state: current_term: 1 leader_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } } }
I20260925 02:56:44.739934 21499 tablet_replica.cc:442] T edadedc44800410296124641774429fa P d1bf63069d8f4d0da52508f00ea11746: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:56:44.740473 21500 tablet_replica.cc:439] T edadedc44800410296124641774429fa P ea99845dd3ed4c30a08935a5123d219d: TxnStatusTablet state changed. Reason: New leader ef3ac5745fdc4b1b994f479bc1c69f60. Latest consensus state: current_term: 1 leader_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } } }
I20260925 02:56:44.740743 21500 tablet_replica.cc:442] T edadedc44800410296124641774429fa P ea99845dd3ed4c30a08935a5123d219d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:56:44.741967 21519 tablet_replica.cc:439] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } } }
I20260925 02:56:44.742290 21519 tablet_replica.cc:442] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:56:44.742444 21520 mvcc.cc:205] Tried to move back new op lower bound from 7333089299343237120 to 7333089298970378240. Current Snapshot: MvccSnapshot[applied={T|T < 7333089299343237120}]
I20260925 02:56:44.747568 21498 tablet_replica.cc:439] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } } }
I20260925 02:56:44.747814 21498 tablet_replica.cc:442] T edadedc44800410296124641774429fa P ef3ac5745fdc4b1b994f479bc1c69f60: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:56:44.748216 21499 tablet_replica.cc:439] T edadedc44800410296124641774429fa P d1bf63069d8f4d0da52508f00ea11746: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } } }
I20260925 02:56:44.748468 21499 tablet_replica.cc:442] T edadedc44800410296124641774429fa P d1bf63069d8f4d0da52508f00ea11746: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:56:44.749064 21524 mvcc.cc:205] Tried to move back new op lower bound from 7333089299343237120 to 7333089298970378240. Current Snapshot: MvccSnapshot[applied={T|T < 7333089299343237120}]
I20260925 02:56:44.756137 21500 tablet_replica.cc:439] T edadedc44800410296124641774429fa P ea99845dd3ed4c30a08935a5123d219d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } } }
I20260925 02:56:44.757704 21500 tablet_replica.cc:442] T edadedc44800410296124641774429fa P ea99845dd3ed4c30a08935a5123d219d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:56:44.763278 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.758294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60910 (local address 127.18.196.193:43197)
0925 02:56:44.758445 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:44.758451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:44.758621 (+ 170us) server_negotiation.cc:407] Connection header received
0925 02:56:44.758824 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:44.758829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:44.758929 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:44.759027 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:44.759489 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.760198 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.761558 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.762001 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.762135 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.762569 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:44.762678 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:44.762789 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:44.762933 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 02:56:44.763008 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:56:44.778316 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.772864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:45849 (local address 127.18.196.193:43197)
0925 02:56:44.773008 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:44.773012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:44.773689 (+ 677us) server_negotiation.cc:407] Connection header received
0925 02:56:44.773871 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:44.773875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:44.773952 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:44.774034 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:44.774462 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.775251 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.776597 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.777024 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.777154 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.777583 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:44.777666 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:44.777769 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:44.777917 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 02:56:44.777983 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":43}
I20260925 02:56:44.778407 21534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.772894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:43197 (local address 127.18.196.193:45849)
0925 02:56:44.773465 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:44.773494 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:44.773765 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:44.774055 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:44.774059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:44.774070 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:56:44.774334 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.774341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.775387 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.775392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.776461 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.776473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.777178 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.777184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.777343 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.777388 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:56:44.777780 (+ 392us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:56:44.777792 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 02:56:44.777899 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 02:56:44.778026 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":3,"thread_start_us":274,"threads_started":1}
I20260925 02:56:44.844238 20985 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:44.844511 20985 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:44.848096 20985 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193: Sending DeleteTablet for 3 replicas of tablet 9b758ea106b14f9db786941b8b10929c
I20260925 02:56:44.849128 21262 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b758ea106b14f9db786941b8b10929c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:44 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:59996
I20260925 02:56:44.849217 21401 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b758ea106b14f9db786941b8b10929c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:44 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:56066
I20260925 02:56:44.850502 21121 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b758ea106b14f9db786941b8b10929c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:44 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:60902
I20260925 02:56:44.851177 20985 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:56:44.851816 20985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:56:44.860086 21537 tablet_replica.cc:330] T 9b758ea106b14f9db786941b8b10929c P ea99845dd3ed4c30a08935a5123d219d: stopping tablet replica
I20260925 02:56:44.860736 21537 raft_consensus.cc:2229] T 9b758ea106b14f9db786941b8b10929c P ea99845dd3ed4c30a08935a5123d219d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:44.861091 21122 tablet_service.cc:1467] Processing CreateTablet for tablet 52524b5c52a14aa59167d6a918b00947 (DEFAULT_TABLE table=test-table [id=1ba8d5d78add490e8b645063a1860d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:44.861173 21537 raft_consensus.cc:2258] T 9b758ea106b14f9db786941b8b10929c P ea99845dd3ed4c30a08935a5123d219d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:44.861578 21122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52524b5c52a14aa59167d6a918b00947. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:44.861936 21402 tablet_service.cc:1467] Processing CreateTablet for tablet 52524b5c52a14aa59167d6a918b00947 (DEFAULT_TABLE table=test-table [id=1ba8d5d78add490e8b645063a1860d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:44.862126 21262 tablet_service.cc:1467] Processing CreateTablet for tablet 52524b5c52a14aa59167d6a918b00947 (DEFAULT_TABLE table=test-table [id=1ba8d5d78add490e8b645063a1860d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:44.862382 21402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52524b5c52a14aa59167d6a918b00947. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:44.862592 21262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52524b5c52a14aa59167d6a918b00947. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:44.863683 21537 ts_tablet_manager.cc:1905] T 9b758ea106b14f9db786941b8b10929c P ea99845dd3ed4c30a08935a5123d219d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:44.865986 21536 tablet_replica.cc:330] T 9b758ea106b14f9db786941b8b10929c P d1bf63069d8f4d0da52508f00ea11746: stopping tablet replica
I20260925 02:56:44.868618 21536 raft_consensus.cc:2229] T 9b758ea106b14f9db786941b8b10929c P d1bf63069d8f4d0da52508f00ea11746 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:44.869029 21536 raft_consensus.cc:2258] T 9b758ea106b14f9db786941b8b10929c P d1bf63069d8f4d0da52508f00ea11746 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:44.871408 21536 ts_tablet_manager.cc:1905] T 9b758ea106b14f9db786941b8b10929c P d1bf63069d8f4d0da52508f00ea11746: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:44.876788 21537 ts_tablet_manager.cc:1918] T 9b758ea106b14f9db786941b8b10929c P ea99845dd3ed4c30a08935a5123d219d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:56:44.882143 21537 log.cc:1196] T 9b758ea106b14f9db786941b8b10929c P ea99845dd3ed4c30a08935a5123d219d: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/9b758ea106b14f9db786941b8b10929c
I20260925 02:56:44.882758 21537 ts_tablet_manager.cc:1939] T 9b758ea106b14f9db786941b8b10929c P ea99845dd3ed4c30a08935a5123d219d: Deleting consensus metadata
I20260925 02:56:44.883157 21539 tablet_bootstrap.cc:492] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746: Bootstrap starting.
I20260925 02:56:44.882002 21540 tablet_bootstrap.cc:492] T 52524b5c52a14aa59167d6a918b00947 P ea99845dd3ed4c30a08935a5123d219d: Bootstrap starting.
I20260925 02:56:44.885339 21539 tablet_bootstrap.cc:654] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:44.885955 21541 tablet_bootstrap.cc:492] T 52524b5c52a14aa59167d6a918b00947 P ef3ac5745fdc4b1b994f479bc1c69f60: Bootstrap starting.
I20260925 02:56:44.886315 21540 tablet_bootstrap.cc:654] T 52524b5c52a14aa59167d6a918b00947 P ea99845dd3ed4c30a08935a5123d219d: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:44.886713 21536 ts_tablet_manager.cc:1918] T 9b758ea106b14f9db786941b8b10929c P d1bf63069d8f4d0da52508f00ea11746: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:56:44.886906 21536 log.cc:1196] T 9b758ea106b14f9db786941b8b10929c P d1bf63069d8f4d0da52508f00ea11746: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/9b758ea106b14f9db786941b8b10929c
I20260925 02:56:44.886917 20973 catalog_manager.cc:4841] TS ea99845dd3ed4c30a08935a5123d219d (127.18.196.194:45877): tablet 9b758ea106b14f9db786941b8b10929c (table test-table [id=6d2d641387fe4eb596d4dcff1fc259cc]) successfully deleted
I20260925 02:56:44.887455 21536 ts_tablet_manager.cc:1939] T 9b758ea106b14f9db786941b8b10929c P d1bf63069d8f4d0da52508f00ea11746: Deleting consensus metadata
I20260925 02:56:44.888288 21541 tablet_bootstrap.cc:654] T 52524b5c52a14aa59167d6a918b00947 P ef3ac5745fdc4b1b994f479bc1c69f60: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:44.889941 20971 catalog_manager.cc:4841] TS d1bf63069d8f4d0da52508f00ea11746 (127.18.196.195:44077): tablet 9b758ea106b14f9db786941b8b10929c (table test-table [id=6d2d641387fe4eb596d4dcff1fc259cc]) successfully deleted
I20260925 02:56:44.890652 21538 tablet_replica.cc:330] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60: stopping tablet replica
I20260925 02:56:44.891129 21538 raft_consensus.cc:2229] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:56:44.891424 21540 tablet_bootstrap.cc:492] T 52524b5c52a14aa59167d6a918b00947 P ea99845dd3ed4c30a08935a5123d219d: No bootstrap required, opened a new log
I20260925 02:56:44.891664 21540 ts_tablet_manager.cc:1397] T 52524b5c52a14aa59167d6a918b00947 P ea99845dd3ed4c30a08935a5123d219d: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260925 02:56:44.891817 21538 raft_consensus.cc:2258] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:44.892882 21540 raft_consensus.cc:348] T 52524b5c52a14aa59167d6a918b00947 P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:44.893301 21539 tablet_bootstrap.cc:492] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746: No bootstrap required, opened a new log
I20260925 02:56:44.893457 21539 ts_tablet_manager.cc:1397] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20260925 02:56:44.893420 21540 raft_consensus.cc:374] T 52524b5c52a14aa59167d6a918b00947 P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:44.893625 21540 raft_consensus.cc:729] T 52524b5c52a14aa59167d6a918b00947 P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea99845dd3ed4c30a08935a5123d219d, State: Initialized, Role: FOLLOWER
I20260925 02:56:44.893913 21538 ts_tablet_manager.cc:1905] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:44.894104 21540 consensus_queue.cc:260] T 52524b5c52a14aa59167d6a918b00947 P ea99845dd3ed4c30a08935a5123d219d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:44.894220 21539 raft_consensus.cc:348] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:44.894459 21539 raft_consensus.cc:374] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:44.894564 21539 raft_consensus.cc:729] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1bf63069d8f4d0da52508f00ea11746, State: Initialized, Role: FOLLOWER
I20260925 02:56:44.894887 21539 consensus_queue.cc:260] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:44.896111 21540 ts_tablet_manager.cc:1428] T 52524b5c52a14aa59167d6a918b00947 P ea99845dd3ed4c30a08935a5123d219d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 02:56:44.896517 21539 ts_tablet_manager.cc:1428] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:56:44.904268 21541 tablet_bootstrap.cc:492] T 52524b5c52a14aa59167d6a918b00947 P ef3ac5745fdc4b1b994f479bc1c69f60: No bootstrap required, opened a new log
I20260925 02:56:44.904449 21541 ts_tablet_manager.cc:1397] T 52524b5c52a14aa59167d6a918b00947 P ef3ac5745fdc4b1b994f479bc1c69f60: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.016s
I20260925 02:56:44.905112 21541 raft_consensus.cc:348] T 52524b5c52a14aa59167d6a918b00947 P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:44.905344 21541 raft_consensus.cc:374] T 52524b5c52a14aa59167d6a918b00947 P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:44.905470 21541 raft_consensus.cc:729] T 52524b5c52a14aa59167d6a918b00947 P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3ac5745fdc4b1b994f479bc1c69f60, State: Initialized, Role: FOLLOWER
I20260925 02:56:44.905768 21541 consensus_queue.cc:260] T 52524b5c52a14aa59167d6a918b00947 P ef3ac5745fdc4b1b994f479bc1c69f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:44.906402 21541 ts_tablet_manager.cc:1428] T 52524b5c52a14aa59167d6a918b00947 P ef3ac5745fdc4b1b994f479bc1c69f60: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 02:56:44.908989 21538 ts_tablet_manager.cc:1918] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:56:44.909155 21538 log.cc:1196] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-0/wal/wals/9b758ea106b14f9db786941b8b10929c
I20260925 02:56:44.909698 21538 ts_tablet_manager.cc:1939] T 9b758ea106b14f9db786941b8b10929c P ef3ac5745fdc4b1b994f479bc1c69f60: Deleting consensus metadata
I20260925 02:56:44.911432 20972 catalog_manager.cc:4841] TS ef3ac5745fdc4b1b994f479bc1c69f60 (127.18.196.193:43197): tablet 9b758ea106b14f9db786941b8b10929c (table test-table [id=6d2d641387fe4eb596d4dcff1fc259cc]) successfully deleted
I20260925 02:56:44.957360 21499 raft_consensus.cc:482] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:56:44.957582 21499 raft_consensus.cc:504] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:44.959211 21499 leader_election.cc:288] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef3ac5745fdc4b1b994f479bc1c69f60 (127.18.196.193:43197), ea99845dd3ed4c30a08935a5123d219d (127.18.196.194:45877)
I20260925 02:56:44.978067 21547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.959830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:48741 (local address 127.18.196.194:45877)
0925 02:56:44.970577 (+ 10747us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:44.970586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:44.971815 (+ 1229us) server_negotiation.cc:407] Connection header received
0925 02:56:44.971909 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:44.971913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:44.971986 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:44.972092 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:44.972868 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.973957 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.975792 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.976873 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.977070 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.977150 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:44.977511 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 02:56:44.977665 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10559,"thread_start_us":10764,"threads_started":1}
I20260925 02:56:44.978060 21545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.959492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:45877 (local address 127.18.196.195:48741)
0925 02:56:44.971588 (+ 12096us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:44.971605 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:44.971707 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:44.972165 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:44.972170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:44.972203 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:44.972642 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.972658 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.974089 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.974095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.975629 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.975644 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.977078 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.977085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.977302 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.977355 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 02:56:44.977467 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 02:56:44.977614 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11976,"mutex_wait_us":46,"thread_start_us":11506,"threads_started":1}
I20260925 02:56:44.978711 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.959810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:60963 (local address 127.18.196.193:43197)
0925 02:56:44.960040 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:44.960049 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:44.971219 (+ 11170us) server_negotiation.cc:407] Connection header received
0925 02:56:44.971577 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:44.971583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:44.971674 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:44.971741 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:44.972489 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.973665 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.976214 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.977519 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.977692 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.977767 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:44.978349 (+ 582us) server_negotiation.cc:299] Negotiation successful
0925 02:56:44.978477 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 02:56:44.979110 21282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52524b5c52a14aa59167d6a918b00947" candidate_uuid: "d1bf63069d8f4d0da52508f00ea11746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea99845dd3ed4c30a08935a5123d219d" is_pre_election: true
I20260925 02:56:44.979375 21546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.959469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:43197 (local address 127.18.196.195:60963)
0925 02:56:44.971108 (+ 11639us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:44.971129 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:44.971387 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:44.971935 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:44.971939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:44.971961 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:44.972318 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.972336 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.973852 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.973859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.976049 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.976067 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.977931 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.977937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.978141 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.978193 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 02:56:44.978974 (+ 781us) client_negotiation.cc:240] Negotiation successful
0925 02:56:44.979078 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11506,"mutex_wait_us":5,"thread_start_us":12029,"threads_started":1}
I20260925 02:56:44.979401 21282 raft_consensus.cc:2454] T 52524b5c52a14aa59167d6a918b00947 P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1bf63069d8f4d0da52508f00ea11746 in term 0.
I20260925 02:56:44.979873 21356 leader_election.cc:302] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1bf63069d8f4d0da52508f00ea11746, ea99845dd3ed4c30a08935a5123d219d; no voters:
I20260925 02:56:44.980285 21499 raft_consensus.cc:2781] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:56:44.980432 21499 raft_consensus.cc:482] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:56:44.980258 21142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52524b5c52a14aa59167d6a918b00947" candidate_uuid: "d1bf63069d8f4d0da52508f00ea11746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" is_pre_election: true
I20260925 02:56:44.980533 21499 raft_consensus.cc:3037] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:44.980698 21142 raft_consensus.cc:2454] T 52524b5c52a14aa59167d6a918b00947 P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1bf63069d8f4d0da52508f00ea11746 in term 0.
I20260925 02:56:44.983053 21499 raft_consensus.cc:504] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:44.983645 21499 leader_election.cc:288] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746 [CANDIDATE]: Term 1 election: Requested vote from peers ef3ac5745fdc4b1b994f479bc1c69f60 (127.18.196.193:43197), ea99845dd3ed4c30a08935a5123d219d (127.18.196.194:45877)
I20260925 02:56:44.984150 21142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52524b5c52a14aa59167d6a918b00947" candidate_uuid: "d1bf63069d8f4d0da52508f00ea11746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60"
I20260925 02:56:44.984156 21282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52524b5c52a14aa59167d6a918b00947" candidate_uuid: "d1bf63069d8f4d0da52508f00ea11746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea99845dd3ed4c30a08935a5123d219d"
I20260925 02:56:44.984354 21142 raft_consensus.cc:3037] T 52524b5c52a14aa59167d6a918b00947 P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:44.984407 21282 raft_consensus.cc:3037] T 52524b5c52a14aa59167d6a918b00947 P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:44.987371 21142 raft_consensus.cc:2454] T 52524b5c52a14aa59167d6a918b00947 P ef3ac5745fdc4b1b994f479bc1c69f60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1bf63069d8f4d0da52508f00ea11746 in term 1.
I20260925 02:56:44.987475 21282 raft_consensus.cc:2454] T 52524b5c52a14aa59167d6a918b00947 P ea99845dd3ed4c30a08935a5123d219d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1bf63069d8f4d0da52508f00ea11746 in term 1.
I20260925 02:56:44.987903 21355 leader_election.cc:302] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1bf63069d8f4d0da52508f00ea11746, ef3ac5745fdc4b1b994f479bc1c69f60; no voters:
I20260925 02:56:44.988312 21499 raft_consensus.cc:2781] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:44.988814 21499 raft_consensus.cc:686] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746 [term 1 LEADER]: Becoming Leader. State: Replica: d1bf63069d8f4d0da52508f00ea11746, State: Running, Role: LEADER
I20260925 02:56:44.989575 21499 consensus_queue.cc:237] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:44.993760 20984 catalog_manager.cc:5495] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1bf63069d8f4d0da52508f00ea11746 (127.18.196.195). New cstate: current_term: 1 leader_uuid: "d1bf63069d8f4d0da52508f00ea11746" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } health_report { overall_health: HEALTHY } } }
I20260925 02:56:45.045089 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.038835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56074 (local address 127.18.196.195:44077)
0925 02:56:45.039982 (+ 1147us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:45.039986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:45.040005 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:56:45.040088 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:45.040091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:45.040160 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:45.040331 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:45.040749 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.041583 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.042872 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.043297 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.043433 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:45.043869 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:45.044449 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:45.044620 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:45.044678 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 02:56:45.044754 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":997,"thread_start_us":854,"threads_started":1}
I20260925 02:56:45.060102 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.054504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:35421 (local address 127.18.196.193:43197)
0925 02:56:45.054660 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:45.054665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:45.055281 (+ 616us) server_negotiation.cc:407] Connection header received
0925 02:56:45.055487 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:45.055490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:45.055562 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:45.055665 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:45.056147 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.056837 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.058159 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.058692 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.058827 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:45.059288 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:45.059403 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:45.059533 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:45.059688 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 02:56:45.059773 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:56:45.060190 21553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.054498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:43197 (local address 127.18.196.195:35421)
0925 02:56:45.055174 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:45.055190 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:45.055339 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:45.055681 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:45.055685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:45.055696 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:56:45.055996 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:45.056004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.056983 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.056988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:45.058015 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:45.058026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.058906 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.058916 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:45.059042 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:45.059095 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:56:45.059545 (+ 450us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:56:45.059560 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 02:56:45.059664 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 02:56:45.059789 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":348,"threads_started":1}
I20260925 02:56:45.069926 21282 raft_consensus.cc:1261] T 52524b5c52a14aa59167d6a918b00947 P ea99845dd3ed4c30a08935a5123d219d [term 1 FOLLOWER]: Refusing update from remote peer d1bf63069d8f4d0da52508f00ea11746: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:45.070106 21142 raft_consensus.cc:1261] T 52524b5c52a14aa59167d6a918b00947 P ef3ac5745fdc4b1b994f479bc1c69f60 [term 1 FOLLOWER]: Refusing update from remote peer d1bf63069d8f4d0da52508f00ea11746: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:45.070603 21548 consensus_queue.cc:1035] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:45.070990 21499 consensus_queue.cc:1035] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:45.078233 21518 mvcc.cc:205] Tried to move back new op lower bound from 7333089300759539712 to 7333089300442484736. Current Snapshot: MvccSnapshot[applied={T|T < 7333089300759539712}]
I20260925 02:56:45.107615 21534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.100537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:44077 (local address 127.18.196.193:43033)
0925 02:56:45.101090 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:45.101111 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:45.101228 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:45.101723 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:45.101728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:45.101742 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:56:45.101968 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:45.101979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.103451 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.103457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:45.105024 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:45.105039 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.106221 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.106227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:45.106432 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:45.106481 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:56:45.106970 (+ 489us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:56:45.106981 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 02:56:45.107071 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 02:56:45.107229 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405}
I20260925 02:56:45.108237 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.100843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:43033 (local address 127.18.196.195:44077)
0925 02:56:45.101459 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:45.101471 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:45.101484 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:56:45.101532 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:45.101535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:45.101591 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:45.101690 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:45.102134 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.103272 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.105205 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.105870 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.106035 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:45.106661 (+ 626us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:45.106788 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:45.106919 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:45.107922 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0925 02:56:45.107998 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502}
W20260925 02:56:45.109302 21520 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 02:56:46.735555 20986 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:46.735780 20986 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:46.739789 20986 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193: Sending DeleteTablet for 3 replicas of tablet 52524b5c52a14aa59167d6a918b00947
I20260925 02:56:46.740593 21121 tablet_service.cc:1514] Processing DeleteTablet for tablet 52524b5c52a14aa59167d6a918b00947 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:46 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:60902
I20260925 02:56:46.740651 21262 tablet_service.cc:1514] Processing DeleteTablet for tablet 52524b5c52a14aa59167d6a918b00947 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:46 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:59996
I20260925 02:56:46.740736 21402 tablet_service.cc:1514] Processing DeleteTablet for tablet 52524b5c52a14aa59167d6a918b00947 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:46 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:56066
I20260925 02:56:46.753252 21583 tablet_replica.cc:330] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746: stopping tablet replica
I20260925 02:56:46.753723 21583 raft_consensus.cc:2229] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:56:46.754736 21583 raft_consensus.cc:2258] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:46.755977 21582 tablet_replica.cc:330] T 52524b5c52a14aa59167d6a918b00947 P ea99845dd3ed4c30a08935a5123d219d: stopping tablet replica
I20260925 02:56:46.756234 21583 ts_tablet_manager.cc:1905] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:46.756325 21582 raft_consensus.cc:2229] T 52524b5c52a14aa59167d6a918b00947 P ea99845dd3ed4c30a08935a5123d219d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:46.756681 21582 raft_consensus.cc:2258] T 52524b5c52a14aa59167d6a918b00947 P ea99845dd3ed4c30a08935a5123d219d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:46.757822 21582 ts_tablet_manager.cc:1905] T 52524b5c52a14aa59167d6a918b00947 P ea99845dd3ed4c30a08935a5123d219d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:46.758976 21581 tablet_replica.cc:330] T 52524b5c52a14aa59167d6a918b00947 P ef3ac5745fdc4b1b994f479bc1c69f60: stopping tablet replica
I20260925 02:56:46.759276 21581 raft_consensus.cc:2229] T 52524b5c52a14aa59167d6a918b00947 P ef3ac5745fdc4b1b994f479bc1c69f60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:46.759582 21581 raft_consensus.cc:2258] T 52524b5c52a14aa59167d6a918b00947 P ef3ac5745fdc4b1b994f479bc1c69f60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:46.760660 21581 ts_tablet_manager.cc:1905] T 52524b5c52a14aa59167d6a918b00947 P ef3ac5745fdc4b1b994f479bc1c69f60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:46.764122 21583 ts_tablet_manager.cc:1918] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:56:46.764325 21583 log.cc:1196] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/52524b5c52a14aa59167d6a918b00947
I20260925 02:56:46.764885 21583 ts_tablet_manager.cc:1939] T 52524b5c52a14aa59167d6a918b00947 P d1bf63069d8f4d0da52508f00ea11746: Deleting consensus metadata
I20260925 02:56:46.765590 21582 ts_tablet_manager.cc:1918] T 52524b5c52a14aa59167d6a918b00947 P ea99845dd3ed4c30a08935a5123d219d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:56:46.765827 21582 log.cc:1196] T 52524b5c52a14aa59167d6a918b00947 P ea99845dd3ed4c30a08935a5123d219d: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/52524b5c52a14aa59167d6a918b00947
I20260925 02:56:46.766179 20971 catalog_manager.cc:4841] TS d1bf63069d8f4d0da52508f00ea11746 (127.18.196.195:44077): tablet 52524b5c52a14aa59167d6a918b00947 (table test-table [id=1ba8d5d78add490e8b645063a1860d9a]) successfully deleted
I20260925 02:56:46.766341 21582 ts_tablet_manager.cc:1939] T 52524b5c52a14aa59167d6a918b00947 P ea99845dd3ed4c30a08935a5123d219d: Deleting consensus metadata
I20260925 02:56:46.767306 21581 ts_tablet_manager.cc:1918] T 52524b5c52a14aa59167d6a918b00947 P ef3ac5745fdc4b1b994f479bc1c69f60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:56:46.767484 21581 log.cc:1196] T 52524b5c52a14aa59167d6a918b00947 P ef3ac5745fdc4b1b994f479bc1c69f60: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-0/wal/wals/52524b5c52a14aa59167d6a918b00947
I20260925 02:56:46.767552 20973 catalog_manager.cc:4841] TS ea99845dd3ed4c30a08935a5123d219d (127.18.196.194:45877): tablet 52524b5c52a14aa59167d6a918b00947 (table test-table [id=1ba8d5d78add490e8b645063a1860d9a]) successfully deleted
I20260925 02:56:46.767966 21581 ts_tablet_manager.cc:1939] T 52524b5c52a14aa59167d6a918b00947 P ef3ac5745fdc4b1b994f479bc1c69f60: Deleting consensus metadata
I20260925 02:56:46.768977 20972 catalog_manager.cc:4841] TS ef3ac5745fdc4b1b994f479bc1c69f60 (127.18.196.193:43197): tablet 52524b5c52a14aa59167d6a918b00947 (table test-table [id=1ba8d5d78add490e8b645063a1860d9a]) successfully deleted
Sep 25 02:56:46 dist-test-slave-8zvk krb5kdc[20941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305006, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:56:46 dist-test-slave-8zvk krb5kdc[20941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305006, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:46.801319 21591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:46.786768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43690 (local address 127.18.196.254:40507)
0925 02:56:46.787354 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:46.787359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:46.787405 (+ 46us) server_negotiation.cc:407] Connection header received
0925 02:56:46.787588 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:46.787592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:46.787671 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:46.787764 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:46.789332 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:46.790091 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:46.791490 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:46.791938 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:46.792039 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:46.796271 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:46.796275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:46.796278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:46.796310 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:46.798789 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:46.799466 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:46.799470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:46.799472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:46.799641 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:46.799947 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:46.799951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:46.799953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:46.800289 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:46.800459 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:46.800782 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 02:56:46.800936 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":301,"threads_started":1}
I20260925 02:56:46.804713 20986 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:56:46.805351 20986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:56:46.811800 21121 tablet_service.cc:1467] Processing CreateTablet for tablet 921866e1c32e4fa78771a75926ae7ff5 (DEFAULT_TABLE table=test-table [id=75c6ad3ed7e1486bb75e06574544335a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:46.811926 21262 tablet_service.cc:1467] Processing CreateTablet for tablet 921866e1c32e4fa78771a75926ae7ff5 (DEFAULT_TABLE table=test-table [id=75c6ad3ed7e1486bb75e06574544335a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:46.812330 21121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921866e1c32e4fa78771a75926ae7ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:46.812363 21262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921866e1c32e4fa78771a75926ae7ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:46.812798 21402 tablet_service.cc:1467] Processing CreateTablet for tablet 921866e1c32e4fa78771a75926ae7ff5 (DEFAULT_TABLE table=test-table [id=75c6ad3ed7e1486bb75e06574544335a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:46.813251 21402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921866e1c32e4fa78771a75926ae7ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:46.819357 21592 tablet_bootstrap.cc:492] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60: Bootstrap starting.
I20260925 02:56:46.820221 21593 tablet_bootstrap.cc:492] T 921866e1c32e4fa78771a75926ae7ff5 P ea99845dd3ed4c30a08935a5123d219d: Bootstrap starting.
I20260925 02:56:46.821686 21592 tablet_bootstrap.cc:654] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:46.822574 21593 tablet_bootstrap.cc:654] T 921866e1c32e4fa78771a75926ae7ff5 P ea99845dd3ed4c30a08935a5123d219d: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:46.825270 21593 tablet_bootstrap.cc:492] T 921866e1c32e4fa78771a75926ae7ff5 P ea99845dd3ed4c30a08935a5123d219d: No bootstrap required, opened a new log
I20260925 02:56:46.825439 21593 ts_tablet_manager.cc:1397] T 921866e1c32e4fa78771a75926ae7ff5 P ea99845dd3ed4c30a08935a5123d219d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:56:46.826366 21594 tablet_bootstrap.cc:492] T 921866e1c32e4fa78771a75926ae7ff5 P d1bf63069d8f4d0da52508f00ea11746: Bootstrap starting.
I20260925 02:56:46.826339 21593 raft_consensus.cc:348] T 921866e1c32e4fa78771a75926ae7ff5 P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:46.826577 21593 raft_consensus.cc:374] T 921866e1c32e4fa78771a75926ae7ff5 P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:46.826670 21593 raft_consensus.cc:729] T 921866e1c32e4fa78771a75926ae7ff5 P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea99845dd3ed4c30a08935a5123d219d, State: Initialized, Role: FOLLOWER
I20260925 02:56:46.826987 21593 consensus_queue.cc:260] T 921866e1c32e4fa78771a75926ae7ff5 P ea99845dd3ed4c30a08935a5123d219d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:46.827606 21592 tablet_bootstrap.cc:492] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60: No bootstrap required, opened a new log
I20260925 02:56:46.827761 21592 ts_tablet_manager.cc:1397] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260925 02:56:46.827831 21593 ts_tablet_manager.cc:1428] T 921866e1c32e4fa78771a75926ae7ff5 P ea99845dd3ed4c30a08935a5123d219d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 02:56:46.828099 21594 tablet_bootstrap.cc:654] T 921866e1c32e4fa78771a75926ae7ff5 P d1bf63069d8f4d0da52508f00ea11746: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:46.828539 21592 raft_consensus.cc:348] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:46.828769 21592 raft_consensus.cc:374] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:46.828866 21592 raft_consensus.cc:729] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3ac5745fdc4b1b994f479bc1c69f60, State: Initialized, Role: FOLLOWER
I20260925 02:56:46.829110 21592 consensus_queue.cc:260] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:46.830631 21592 ts_tablet_manager.cc:1428] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 02:56:46.830812 21594 tablet_bootstrap.cc:492] T 921866e1c32e4fa78771a75926ae7ff5 P d1bf63069d8f4d0da52508f00ea11746: No bootstrap required, opened a new log
I20260925 02:56:46.830994 21594 ts_tablet_manager.cc:1397] T 921866e1c32e4fa78771a75926ae7ff5 P d1bf63069d8f4d0da52508f00ea11746: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260925 02:56:46.831985 21594 raft_consensus.cc:348] T 921866e1c32e4fa78771a75926ae7ff5 P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:46.832229 21594 raft_consensus.cc:374] T 921866e1c32e4fa78771a75926ae7ff5 P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:46.832340 21594 raft_consensus.cc:729] T 921866e1c32e4fa78771a75926ae7ff5 P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1bf63069d8f4d0da52508f00ea11746, State: Initialized, Role: FOLLOWER
I20260925 02:56:46.832602 21594 consensus_queue.cc:260] T 921866e1c32e4fa78771a75926ae7ff5 P d1bf63069d8f4d0da52508f00ea11746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:46.833242 21594 ts_tablet_manager.cc:1428] T 921866e1c32e4fa78771a75926ae7ff5 P d1bf63069d8f4d0da52508f00ea11746: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 02:56:46.879679 21561 raft_consensus.cc:482] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:56:46.879952 21561 raft_consensus.cc:504] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:46.880782 21561 leader_election.cc:288] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea99845dd3ed4c30a08935a5123d219d (127.18.196.194:45877), d1bf63069d8f4d0da52508f00ea11746 (127.18.196.195:44077)
I20260925 02:56:46.881207 21282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921866e1c32e4fa78771a75926ae7ff5" candidate_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea99845dd3ed4c30a08935a5123d219d" is_pre_election: true
I20260925 02:56:46.881389 21421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921866e1c32e4fa78771a75926ae7ff5" candidate_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1bf63069d8f4d0da52508f00ea11746" is_pre_election: true
I20260925 02:56:46.881479 21282 raft_consensus.cc:2454] T 921866e1c32e4fa78771a75926ae7ff5 P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef3ac5745fdc4b1b994f479bc1c69f60 in term 0.
I20260925 02:56:46.881696 21421 raft_consensus.cc:2454] T 921866e1c32e4fa78771a75926ae7ff5 P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef3ac5745fdc4b1b994f479bc1c69f60 in term 0.
I20260925 02:56:46.882038 21076 leader_election.cc:302] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea99845dd3ed4c30a08935a5123d219d, ef3ac5745fdc4b1b994f479bc1c69f60; no voters:
I20260925 02:56:46.882365 21561 raft_consensus.cc:2781] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:56:46.882503 21561 raft_consensus.cc:482] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:56:46.882596 21561 raft_consensus.cc:3037] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:46.885053 21561 raft_consensus.cc:504] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:46.885732 21561 leader_election.cc:288] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60 [CANDIDATE]: Term 1 election: Requested vote from peers ea99845dd3ed4c30a08935a5123d219d (127.18.196.194:45877), d1bf63069d8f4d0da52508f00ea11746 (127.18.196.195:44077)
I20260925 02:56:46.886237 21282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921866e1c32e4fa78771a75926ae7ff5" candidate_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea99845dd3ed4c30a08935a5123d219d"
I20260925 02:56:46.886299 21421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921866e1c32e4fa78771a75926ae7ff5" candidate_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1bf63069d8f4d0da52508f00ea11746"
I20260925 02:56:46.886488 21282 raft_consensus.cc:3037] T 921866e1c32e4fa78771a75926ae7ff5 P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:46.886562 21421 raft_consensus.cc:3037] T 921866e1c32e4fa78771a75926ae7ff5 P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:46.888731 21282 raft_consensus.cc:2454] T 921866e1c32e4fa78771a75926ae7ff5 P ea99845dd3ed4c30a08935a5123d219d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef3ac5745fdc4b1b994f479bc1c69f60 in term 1.
I20260925 02:56:46.889158 21421 raft_consensus.cc:2454] T 921866e1c32e4fa78771a75926ae7ff5 P d1bf63069d8f4d0da52508f00ea11746 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef3ac5745fdc4b1b994f479bc1c69f60 in term 1.
I20260925 02:56:46.889175 21076 leader_election.cc:302] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea99845dd3ed4c30a08935a5123d219d, ef3ac5745fdc4b1b994f479bc1c69f60; no voters:
I20260925 02:56:46.889539 21561 raft_consensus.cc:2781] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:46.889715 21561 raft_consensus.cc:686] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60 [term 1 LEADER]: Becoming Leader. State: Replica: ef3ac5745fdc4b1b994f479bc1c69f60, State: Running, Role: LEADER
I20260925 02:56:46.890023 21561 consensus_queue.cc:237] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:46.892958 20984 catalog_manager.cc:5495] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef3ac5745fdc4b1b994f479bc1c69f60 (127.18.196.193). New cstate: current_term: 1 leader_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } health_report { overall_health: UNKNOWN } } }
I20260925 02:56:46.932897 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:46.927368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60924 (local address 127.18.196.193:43197)
0925 02:56:46.928123 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:46.928131 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:46.928160 (+ 29us) server_negotiation.cc:407] Connection header received
0925 02:56:46.928239 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:46.928245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:46.928347 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:46.928466 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:46.928942 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:46.929989 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:46.931245 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:46.931641 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:46.931763 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:46.932182 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:46.932285 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:46.932351 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:46.932490 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 02:56:46.932551 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":372,"threads_started":1}
I20260925 02:56:46.942276 21421 raft_consensus.cc:1261] T 921866e1c32e4fa78771a75926ae7ff5 P d1bf63069d8f4d0da52508f00ea11746 [term 1 FOLLOWER]: Refusing update from remote peer ef3ac5745fdc4b1b994f479bc1c69f60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:46.942785 21561 consensus_queue.cc:1035] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:46.942914 21282 raft_consensus.cc:1261] T 921866e1c32e4fa78771a75926ae7ff5 P ea99845dd3ed4c30a08935a5123d219d [term 1 FOLLOWER]: Refusing update from remote peer ef3ac5745fdc4b1b994f479bc1c69f60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:46.943459 21561 consensus_queue.cc:1035] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:46.998950 20986 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:46.999212 20986 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:47.001938 20986 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193: Sending DeleteTablet for 3 replicas of tablet 921866e1c32e4fa78771a75926ae7ff5
I20260925 02:56:47.002718 21121 tablet_service.cc:1514] Processing DeleteTablet for tablet 921866e1c32e4fa78771a75926ae7ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:46 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:60902
I20260925 02:56:47.002897 21262 tablet_service.cc:1514] Processing DeleteTablet for tablet 921866e1c32e4fa78771a75926ae7ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:46 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:59996
I20260925 02:56:47.002933 21402 tablet_service.cc:1514] Processing DeleteTablet for tablet 921866e1c32e4fa78771a75926ae7ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:46 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:56066
I20260925 02:56:47.003204 21583 tablet_replica.cc:330] T 921866e1c32e4fa78771a75926ae7ff5 P d1bf63069d8f4d0da52508f00ea11746: stopping tablet replica
I20260925 02:56:47.003203 21581 tablet_replica.cc:330] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60: stopping tablet replica
I20260925 02:56:47.003273 21582 tablet_replica.cc:330] T 921866e1c32e4fa78771a75926ae7ff5 P ea99845dd3ed4c30a08935a5123d219d: stopping tablet replica
I20260925 02:56:47.003427 21583 raft_consensus.cc:2229] T 921866e1c32e4fa78771a75926ae7ff5 P d1bf63069d8f4d0da52508f00ea11746 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:47.003543 21581 raft_consensus.cc:2229] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:56:47.003610 21582 raft_consensus.cc:2229] T 921866e1c32e4fa78771a75926ae7ff5 P ea99845dd3ed4c30a08935a5123d219d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:47.003682 21583 raft_consensus.cc:2258] T 921866e1c32e4fa78771a75926ae7ff5 P d1bf63069d8f4d0da52508f00ea11746 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:47.003818 21582 raft_consensus.cc:2258] T 921866e1c32e4fa78771a75926ae7ff5 P ea99845dd3ed4c30a08935a5123d219d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:47.003921 21581 raft_consensus.cc:2258] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:47.003918 20986 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:56:47.004616 20986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:56:47.005129 21583 ts_tablet_manager.cc:1905] T 921866e1c32e4fa78771a75926ae7ff5 P d1bf63069d8f4d0da52508f00ea11746: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:47.005141 21582 ts_tablet_manager.cc:1905] T 921866e1c32e4fa78771a75926ae7ff5 P ea99845dd3ed4c30a08935a5123d219d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:47.006819 21581 ts_tablet_manager.cc:1905] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:47.011806 21583 ts_tablet_manager.cc:1918] T 921866e1c32e4fa78771a75926ae7ff5 P d1bf63069d8f4d0da52508f00ea11746: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:56:47.011991 21583 log.cc:1196] T 921866e1c32e4fa78771a75926ae7ff5 P d1bf63069d8f4d0da52508f00ea11746: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/921866e1c32e4fa78771a75926ae7ff5
I20260925 02:56:47.012420 21583 ts_tablet_manager.cc:1939] T 921866e1c32e4fa78771a75926ae7ff5 P d1bf63069d8f4d0da52508f00ea11746: Deleting consensus metadata
I20260925 02:56:47.012733 21262 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3ad2519fe041168553175cd9d782fb (DEFAULT_TABLE table=test-table [id=98cfd4c459644dda96cadb6497fb8b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:47.013175 21262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d3ad2519fe041168553175cd9d782fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:47.013233 21581 ts_tablet_manager.cc:1918] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:56:47.013422 21581 log.cc:1196] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-0/wal/wals/921866e1c32e4fa78771a75926ae7ff5
I20260925 02:56:47.013907 21581 ts_tablet_manager.cc:1939] T 921866e1c32e4fa78771a75926ae7ff5 P ef3ac5745fdc4b1b994f479bc1c69f60: Deleting consensus metadata
I20260925 02:56:47.014626 21402 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3ad2519fe041168553175cd9d782fb (DEFAULT_TABLE table=test-table [id=98cfd4c459644dda96cadb6497fb8b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:47.015022 21402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d3ad2519fe041168553175cd9d782fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:47.016657 21582 ts_tablet_manager.cc:1918] T 921866e1c32e4fa78771a75926ae7ff5 P ea99845dd3ed4c30a08935a5123d219d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:56:47.016832 21582 log.cc:1196] T 921866e1c32e4fa78771a75926ae7ff5 P ea99845dd3ed4c30a08935a5123d219d: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/921866e1c32e4fa78771a75926ae7ff5
I20260925 02:56:47.017297 21582 ts_tablet_manager.cc:1939] T 921866e1c32e4fa78771a75926ae7ff5 P ea99845dd3ed4c30a08935a5123d219d: Deleting consensus metadata
I20260925 02:56:47.013911 21121 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3ad2519fe041168553175cd9d782fb (DEFAULT_TABLE table=test-table [id=98cfd4c459644dda96cadb6497fb8b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:47.018352 21121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d3ad2519fe041168553175cd9d782fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:47.018299 20973 catalog_manager.cc:4841] TS ea99845dd3ed4c30a08935a5123d219d (127.18.196.194:45877): tablet 921866e1c32e4fa78771a75926ae7ff5 (table test-table [id=75c6ad3ed7e1486bb75e06574544335a]) successfully deleted
I20260925 02:56:47.020433 21593 tablet_bootstrap.cc:492] T 2d3ad2519fe041168553175cd9d782fb P ea99845dd3ed4c30a08935a5123d219d: Bootstrap starting.
I20260925 02:56:47.020859 20972 catalog_manager.cc:4841] TS ef3ac5745fdc4b1b994f479bc1c69f60 (127.18.196.193:43197): tablet 921866e1c32e4fa78771a75926ae7ff5 (table test-table [id=75c6ad3ed7e1486bb75e06574544335a]) successfully deleted
I20260925 02:56:47.021515 21594 tablet_bootstrap.cc:492] T 2d3ad2519fe041168553175cd9d782fb P d1bf63069d8f4d0da52508f00ea11746: Bootstrap starting.
I20260925 02:56:47.021888 20971 catalog_manager.cc:4841] TS d1bf63069d8f4d0da52508f00ea11746 (127.18.196.195:44077): tablet 921866e1c32e4fa78771a75926ae7ff5 (table test-table [id=75c6ad3ed7e1486bb75e06574544335a]) successfully deleted
I20260925 02:56:47.022230 21593 tablet_bootstrap.cc:654] T 2d3ad2519fe041168553175cd9d782fb P ea99845dd3ed4c30a08935a5123d219d: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:47.023125 21594 tablet_bootstrap.cc:654] T 2d3ad2519fe041168553175cd9d782fb P d1bf63069d8f4d0da52508f00ea11746: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:47.024277 21592 tablet_bootstrap.cc:492] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60: Bootstrap starting.
I20260925 02:56:47.025849 21593 tablet_bootstrap.cc:492] T 2d3ad2519fe041168553175cd9d782fb P ea99845dd3ed4c30a08935a5123d219d: No bootstrap required, opened a new log
I20260925 02:56:47.026049 21593 ts_tablet_manager.cc:1397] T 2d3ad2519fe041168553175cd9d782fb P ea99845dd3ed4c30a08935a5123d219d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:56:47.026160 21592 tablet_bootstrap.cc:654] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:47.027012 21594 tablet_bootstrap.cc:492] T 2d3ad2519fe041168553175cd9d782fb P d1bf63069d8f4d0da52508f00ea11746: No bootstrap required, opened a new log
I20260925 02:56:47.026898 21593 raft_consensus.cc:348] T 2d3ad2519fe041168553175cd9d782fb P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:47.027192 21593 raft_consensus.cc:374] T 2d3ad2519fe041168553175cd9d782fb P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:47.027287 21593 raft_consensus.cc:729] T 2d3ad2519fe041168553175cd9d782fb P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea99845dd3ed4c30a08935a5123d219d, State: Initialized, Role: FOLLOWER
I20260925 02:56:47.027326 21594 ts_tablet_manager.cc:1397] T 2d3ad2519fe041168553175cd9d782fb P d1bf63069d8f4d0da52508f00ea11746: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:56:47.027586 21593 consensus_queue.cc:260] T 2d3ad2519fe041168553175cd9d782fb P ea99845dd3ed4c30a08935a5123d219d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:47.028352 21593 ts_tablet_manager.cc:1428] T 2d3ad2519fe041168553175cd9d782fb P ea99845dd3ed4c30a08935a5123d219d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 02:56:47.028264 21594 raft_consensus.cc:348] T 2d3ad2519fe041168553175cd9d782fb P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:47.028483 21594 raft_consensus.cc:374] T 2d3ad2519fe041168553175cd9d782fb P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:47.028581 21594 raft_consensus.cc:729] T 2d3ad2519fe041168553175cd9d782fb P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1bf63069d8f4d0da52508f00ea11746, State: Initialized, Role: FOLLOWER
I20260925 02:56:47.028873 21594 consensus_queue.cc:260] T 2d3ad2519fe041168553175cd9d782fb P d1bf63069d8f4d0da52508f00ea11746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:47.029129 21592 tablet_bootstrap.cc:492] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60: No bootstrap required, opened a new log
I20260925 02:56:47.029295 21592 ts_tablet_manager.cc:1397] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 02:56:47.030030 21594 ts_tablet_manager.cc:1428] T 2d3ad2519fe041168553175cd9d782fb P d1bf63069d8f4d0da52508f00ea11746: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:56:47.030030 21592 raft_consensus.cc:348] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:47.030344 21592 raft_consensus.cc:374] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:47.030472 21592 raft_consensus.cc:729] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3ac5745fdc4b1b994f479bc1c69f60, State: Initialized, Role: FOLLOWER
I20260925 02:56:47.030736 21592 consensus_queue.cc:260] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:47.031905 21592 ts_tablet_manager.cc:1428] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 02:56:47.416489 21605 raft_consensus.cc:482] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:56:47.416786 21605 raft_consensus.cc:504] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:47.417594 21605 leader_election.cc:288] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea99845dd3ed4c30a08935a5123d219d (127.18.196.194:45877), d1bf63069d8f4d0da52508f00ea11746 (127.18.196.195:44077)
I20260925 02:56:47.418180 21282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3ad2519fe041168553175cd9d782fb" candidate_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea99845dd3ed4c30a08935a5123d219d" is_pre_election: true
I20260925 02:56:47.418187 21422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3ad2519fe041168553175cd9d782fb" candidate_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1bf63069d8f4d0da52508f00ea11746" is_pre_election: true
I20260925 02:56:47.418447 21282 raft_consensus.cc:2454] T 2d3ad2519fe041168553175cd9d782fb P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef3ac5745fdc4b1b994f479bc1c69f60 in term 0.
I20260925 02:56:47.418501 21422 raft_consensus.cc:2454] T 2d3ad2519fe041168553175cd9d782fb P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef3ac5745fdc4b1b994f479bc1c69f60 in term 0.
I20260925 02:56:47.418964 21076 leader_election.cc:302] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea99845dd3ed4c30a08935a5123d219d, ef3ac5745fdc4b1b994f479bc1c69f60; no voters:
I20260925 02:56:47.419237 21605 raft_consensus.cc:2781] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:56:47.419425 21605 raft_consensus.cc:482] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:56:47.419600 21605 raft_consensus.cc:3037] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:47.422045 21605 raft_consensus.cc:504] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:47.422631 21605 leader_election.cc:288] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60 [CANDIDATE]: Term 1 election: Requested vote from peers ea99845dd3ed4c30a08935a5123d219d (127.18.196.194:45877), d1bf63069d8f4d0da52508f00ea11746 (127.18.196.195:44077)
I20260925 02:56:47.423054 21282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3ad2519fe041168553175cd9d782fb" candidate_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea99845dd3ed4c30a08935a5123d219d"
I20260925 02:56:47.423204 21422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3ad2519fe041168553175cd9d782fb" candidate_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1bf63069d8f4d0da52508f00ea11746"
I20260925 02:56:47.423285 21282 raft_consensus.cc:3037] T 2d3ad2519fe041168553175cd9d782fb P ea99845dd3ed4c30a08935a5123d219d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:47.423457 21422 raft_consensus.cc:3037] T 2d3ad2519fe041168553175cd9d782fb P d1bf63069d8f4d0da52508f00ea11746 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:47.425587 21422 raft_consensus.cc:2454] T 2d3ad2519fe041168553175cd9d782fb P d1bf63069d8f4d0da52508f00ea11746 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef3ac5745fdc4b1b994f479bc1c69f60 in term 1.
I20260925 02:56:47.425858 21282 raft_consensus.cc:2454] T 2d3ad2519fe041168553175cd9d782fb P ea99845dd3ed4c30a08935a5123d219d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef3ac5745fdc4b1b994f479bc1c69f60 in term 1.
I20260925 02:56:47.425987 21074 leader_election.cc:302] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1bf63069d8f4d0da52508f00ea11746, ef3ac5745fdc4b1b994f479bc1c69f60; no voters:
I20260925 02:56:47.426281 21605 raft_consensus.cc:2781] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:47.426431 21605 raft_consensus.cc:686] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60 [term 1 LEADER]: Becoming Leader. State: Replica: ef3ac5745fdc4b1b994f479bc1c69f60, State: Running, Role: LEADER
I20260925 02:56:47.426689 21605 consensus_queue.cc:237] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } }
I20260925 02:56:47.429315 20986 catalog_manager.cc:5495] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef3ac5745fdc4b1b994f479bc1c69f60 (127.18.196.193). New cstate: current_term: 1 leader_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3ac5745fdc4b1b994f479bc1c69f60" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 } health_report { overall_health: UNKNOWN } } }
I20260925 02:56:47.507705 21422 raft_consensus.cc:1261] T 2d3ad2519fe041168553175cd9d782fb P d1bf63069d8f4d0da52508f00ea11746 [term 1 FOLLOWER]: Refusing update from remote peer ef3ac5745fdc4b1b994f479bc1c69f60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:47.507714 21282 raft_consensus.cc:1261] T 2d3ad2519fe041168553175cd9d782fb P ea99845dd3ed4c30a08935a5123d219d [term 1 FOLLOWER]: Refusing update from remote peer ef3ac5745fdc4b1b994f479bc1c69f60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:47.508301 21614 consensus_queue.cc:1035] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1bf63069d8f4d0da52508f00ea11746" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 44077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:47.508569 21605 consensus_queue.cc:1035] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea99845dd3ed4c30a08935a5123d219d" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 45877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 02:56:47.529299 21610 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 02:56:49.129204 20984 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:49.129522 20984 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:49.133581 20984 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b5e1c717ad045279c8ba63b64375193: Sending DeleteTablet for 3 replicas of tablet 2d3ad2519fe041168553175cd9d782fb
I20260925 02:56:49.134315 21121 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d3ad2519fe041168553175cd9d782fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:49 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:60902
I20260925 02:56:49.134339 21262 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d3ad2519fe041168553175cd9d782fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:49 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:59996
I20260925 02:56:49.134574 21402 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d3ad2519fe041168553175cd9d782fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:49 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:56066
I20260925 02:56:49.135301 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 21057
I20260925 02:56:49.137643 21645 tablet_replica.cc:330] T 2d3ad2519fe041168553175cd9d782fb P d1bf63069d8f4d0da52508f00ea11746: stopping tablet replica
I20260925 02:56:49.137970 21645 raft_consensus.cc:2229] T 2d3ad2519fe041168553175cd9d782fb P d1bf63069d8f4d0da52508f00ea11746 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:49.138388 21645 raft_consensus.cc:2258] T 2d3ad2519fe041168553175cd9d782fb P d1bf63069d8f4d0da52508f00ea11746 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:49.140190 21645 ts_tablet_manager.cc:1905] T 2d3ad2519fe041168553175cd9d782fb P d1bf63069d8f4d0da52508f00ea11746: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:49.147109 21644 tablet_replica.cc:330] T 2d3ad2519fe041168553175cd9d782fb P ea99845dd3ed4c30a08935a5123d219d: stopping tablet replica
I20260925 02:56:49.147322 21644 raft_consensus.cc:2229] T 2d3ad2519fe041168553175cd9d782fb P ea99845dd3ed4c30a08935a5123d219d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:49.147608 21644 raft_consensus.cc:2258] T 2d3ad2519fe041168553175cd9d782fb P ea99845dd3ed4c30a08935a5123d219d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:49.148192 21643 tablet_replica.cc:330] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60: stopping tablet replica
I20260925 02:56:49.148440 21643 raft_consensus.cc:2229] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:56:49.148598 21644 ts_tablet_manager.cc:1905] T 2d3ad2519fe041168553175cd9d782fb P ea99845dd3ed4c30a08935a5123d219d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:49.148860 21643 raft_consensus.cc:2258] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:49.149649 21645 ts_tablet_manager.cc:1918] T 2d3ad2519fe041168553175cd9d782fb P d1bf63069d8f4d0da52508f00ea11746: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:56:49.149686 21643 ts_tablet_manager.cc:1905] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:49.149856 21645 log.cc:1196] T 2d3ad2519fe041168553175cd9d782fb P d1bf63069d8f4d0da52508f00ea11746: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/2d3ad2519fe041168553175cd9d782fb
I20260925 02:56:49.150358 21645 ts_tablet_manager.cc:1939] T 2d3ad2519fe041168553175cd9d782fb P d1bf63069d8f4d0da52508f00ea11746: Deleting consensus metadata
I20260925 02:56:49.151436 20971 catalog_manager.cc:4841] TS d1bf63069d8f4d0da52508f00ea11746 (127.18.196.195:44077): tablet 2d3ad2519fe041168553175cd9d782fb (table test-table [id=98cfd4c459644dda96cadb6497fb8b92]) successfully deleted
I20260925 02:56:49.154650 21644 ts_tablet_manager.cc:1918] T 2d3ad2519fe041168553175cd9d782fb P ea99845dd3ed4c30a08935a5123d219d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:56:49.154778 21644 log.cc:1196] T 2d3ad2519fe041168553175cd9d782fb P ea99845dd3ed4c30a08935a5123d219d: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/2d3ad2519fe041168553175cd9d782fb
I20260925 02:56:49.155263 21644 ts_tablet_manager.cc:1939] T 2d3ad2519fe041168553175cd9d782fb P ea99845dd3ed4c30a08935a5123d219d: Deleting consensus metadata
I20260925 02:56:49.155706 21643 ts_tablet_manager.cc:1918] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:56:49.155802 21643 log.cc:1196] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304987644417-19219-0/minicluster-data/ts-0/wal/wals/2d3ad2519fe041168553175cd9d782fb
I20260925 02:56:49.156140 20973 catalog_manager.cc:4841] TS ea99845dd3ed4c30a08935a5123d219d (127.18.196.194:45877): tablet 2d3ad2519fe041168553175cd9d782fb (table test-table [id=98cfd4c459644dda96cadb6497fb8b92]) successfully deleted
I20260925 02:56:49.156258 21643 ts_tablet_manager.cc:1939] T 2d3ad2519fe041168553175cd9d782fb P ef3ac5745fdc4b1b994f479bc1c69f60: Deleting consensus metadata
I20260925 02:56:49.157162 20972 catalog_manager.cc:4841] TS ef3ac5745fdc4b1b994f479bc1c69f60 (127.18.196.193:43197): tablet 2d3ad2519fe041168553175cd9d782fb (table test-table [id=98cfd4c459644dda96cadb6497fb8b92]) successfully deleted
Sep 25 02:56:49 dist-test-slave-8zvk krb5kdc[20941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305006, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.196.193@KRBTEST.COM
I20260925 02:56:49.165114 21647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:49.137108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60940 (local address 127.18.196.193:43197)
0925 02:56:49.145007 (+ 7899us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:49.145016 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:49.145916 (+ 900us) server_negotiation.cc:407] Connection header received
0925 02:56:49.146135 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:49.146144 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:49.146259 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:49.146381 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:49.148344 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:49.149442 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:49.151292 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:49.152030 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:49.152185 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:49.159781 (+ 7596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:49.159795 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:49.159811 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:49.159843 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:49.162759 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:49.163407 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:49.163414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:49.163416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:49.163625 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:49.163951 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:49.163954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:49.163956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:49.164288 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:49.164486 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:49.164731 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 02:56:49.164878 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7708,"thread_start_us":7541,"threads_started":1}
W20260925 02:56:49.165710 21187 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60940
W20260925 02:56:49.166277 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 21057: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:56:49.166489 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 21057
I20260925 02:56:49.193518 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 21196
Sep 25 02:56:49 dist-test-slave-8zvk krb5kdc[20941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305006, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.196.194@KRBTEST.COM
I20260925 02:56:49.208626 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:49.194259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60002 (local address 127.18.196.194:45877)
0925 02:56:49.194767 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:49.194771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:49.194786 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:56:49.194881 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:49.194888 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:49.194957 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:49.195044 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:49.196343 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:49.197112 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:49.198295 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:49.198677 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:49.198764 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:49.203046 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:49.203058 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:49.203072 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:49.203107 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:49.205757 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:49.206380 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:49.206387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:49.206390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:49.206584 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:49.207061 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:49.207066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:49.207068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:49.207407 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:49.207709 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:49.208065 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 02:56:49.208309 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":251,"threads_started":1}
W20260925 02:56:49.209244 21328 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60002
W20260925 02:56:49.209848 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 21196: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:56:49.209975 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 21196
I20260925 02:56:49.230875 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 21337
Sep 25 02:56:49 dist-test-slave-8zvk krb5kdc[20941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305006, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.196.195@KRBTEST.COM
I20260925 02:56:49.245796 21649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:49.231594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56088 (local address 127.18.196.195:44077)
0925 02:56:49.232153 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:49.232157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:49.232173 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:56:49.232227 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:49.232231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:49.232289 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:49.232379 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:49.233594 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:49.234308 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:49.235578 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:49.235971 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:49.236062 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:49.240216 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:49.240230 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:49.240246 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:49.240283 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:49.243321 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:49.243908 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:49.243915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:49.243917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:49.244062 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:49.244485 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:49.244489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:49.244491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:49.244832 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:49.245050 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:49.245356 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 02:56:49.245529 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":317,"threads_started":1}
W20260925 02:56:49.246488 21468 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56088
W20260925 02:56:49.247092 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 21337: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:56:49.247234 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 21337
I20260925 02:56:49.269742 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 20954
I20260925 02:56:49.270905 21026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:56:49.386490 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 20954
2026-09-25T02:56:49Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8529 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:49 dist-test-slave-8zvk krb5kdc[21653](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:49 dist-test-slave-8zvk krb5kdc[21653](info): set up 2 sockets
Sep 25 02:56:49 dist-test-slave-8zvk krb5kdc[21653](info): commencing operation
krb5kdc: starting...
W20260925 02:56:50.470194 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 02:56:50 dist-test-slave-8zvk krb5kdc[21653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 02:56:50.656594 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:33767
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:33767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:50.970413 21666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:50.970700 21666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:50.970789 21666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:50.978045 21666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:50.978158 21666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:50.978205 21666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:50.978262 21666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:50.978307 21666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:50.988303 21666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:33767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:33767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:50.989562 21666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:50.990806 21666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:51.002823 21671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:51.002935 21672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:51.002884 21674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:51.003326 21666 server_base.cc:1034] running on GCE node
I20260925 02:56:51.004185 21666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:51.005512 21666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:51.006754 21666 hybrid_clock.cc:648] HybridClock initialized: now 1790305011006721 us; error 48 us; skew 500 ppm
I20260925 02:56:51.007140 21666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:51 dist-test-slave-8zvk krb5kdc[21653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305011, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:51.010108 21666 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 02:56:51.011719 21666 webserver.cc:460] Webserver started at http://127.18.196.254:36371/ using document root <none> and password file <none>
I20260925 02:56:51.012329 21666 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:51.012426 21666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:51.012756 21666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:51.015625 21666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "29aeda7afdee40739eb93a5849bdc240"
format_stamp: "Formatted at 2026-09-25 02:56:51 on dist-test-slave-8zvk"
server_key: "d52330eb7af07f2a30db8d7a675117c8"
server_key_iv: "af221c0d07cac09492c98e54e1af9d77"
server_key_version: "encryptionkey@0"
I20260925 02:56:51.016307 21666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "29aeda7afdee40739eb93a5849bdc240"
format_stamp: "Formatted at 2026-09-25 02:56:51 on dist-test-slave-8zvk"
server_key: "d52330eb7af07f2a30db8d7a675117c8"
server_key_iv: "af221c0d07cac09492c98e54e1af9d77"
server_key_version: "encryptionkey@0"
I20260925 02:56:51.021193 21666 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:56:51.024442 21681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:51.025642 21666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:56:51.025821 21666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "29aeda7afdee40739eb93a5849bdc240"
format_stamp: "Formatted at 2026-09-25 02:56:51 on dist-test-slave-8zvk"
server_key: "d52330eb7af07f2a30db8d7a675117c8"
server_key_iv: "af221c0d07cac09492c98e54e1af9d77"
server_key_version: "encryptionkey@0"
I20260925 02:56:51.026007 21666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:51.037526 21666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:51.041421 21666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:51.041710 21666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:51.060062 21666 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:33767
I20260925 02:56:51.060081 21733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:33767 every 8 connection(s)
I20260925 02:56:51.061440 21666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:51.063882 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 21666
I20260925 02:56:51.064138 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:56:51.064921 19219 external_mini_cluster.cc:1428] Setting key ff091ac150da55001af1a7504d7b3de2
I20260925 02:56:51.065743 21734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:51.079850 21734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29aeda7afdee40739eb93a5849bdc240: Bootstrap starting.
Sep 25 02:56:51 dist-test-slave-8zvk krb5kdc[21653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:51.083937 21734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29aeda7afdee40739eb93a5849bdc240: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:51.085282 21734 log.cc:824] T 00000000000000000000000000000000 P 29aeda7afdee40739eb93a5849bdc240: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:51.088778 21734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29aeda7afdee40739eb93a5849bdc240: No bootstrap required, opened a new log
I20260925 02:56:51.089969 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:51.067051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58188 (local address 127.18.196.254:33767)
0925 02:56:51.067787 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:51.067797 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:51.067833 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:56:51.070124 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:51.070145 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:51.070506 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:51.070821 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:51.072637 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:51.073876 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:51.075650 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:51.076309 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:51.076463 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:51.081218 (+ 4755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:51.081229 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:51.081249 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:51.081329 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:51.085408 (+ 4079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:51.086230 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:51.086237 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:51.086245 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:51.086409 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:51.086880 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:51.086888 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:51.086893 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:51.087541 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:51.087818 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:51.088232 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 02:56:51.088650 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":339,"threads_started":1}
I20260925 02:56:51.097057 21734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29aeda7afdee40739eb93a5849bdc240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29aeda7afdee40739eb93a5849bdc240" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33767 } }
I20260925 02:56:51.097445 21734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29aeda7afdee40739eb93a5849bdc240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:51.097518 21734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29aeda7afdee40739eb93a5849bdc240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29aeda7afdee40739eb93a5849bdc240, State: Initialized, Role: FOLLOWER
I20260925 02:56:51.097988 21734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29aeda7afdee40739eb93a5849bdc240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29aeda7afdee40739eb93a5849bdc240" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33767 } }
I20260925 02:56:51.098184 21734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29aeda7afdee40739eb93a5849bdc240 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:51.098284 21734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29aeda7afdee40739eb93a5849bdc240 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:51.098402 21734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29aeda7afdee40739eb93a5849bdc240 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:51.100502 21734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29aeda7afdee40739eb93a5849bdc240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29aeda7afdee40739eb93a5849bdc240" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33767 } }
I20260925 02:56:51.100934 21734 leader_election.cc:302] T 00000000000000000000000000000000 P 29aeda7afdee40739eb93a5849bdc240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29aeda7afdee40739eb93a5849bdc240; no voters:
I20260925 02:56:51.101460 21734 leader_election.cc:288] T 00000000000000000000000000000000 P 29aeda7afdee40739eb93a5849bdc240 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:51.101599 21739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29aeda7afdee40739eb93a5849bdc240 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:51.102281 21739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29aeda7afdee40739eb93a5849bdc240 [term 1 LEADER]: Becoming Leader. State: Replica: 29aeda7afdee40739eb93a5849bdc240, State: Running, Role: LEADER
I20260925 02:56:51.102875 21739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29aeda7afdee40739eb93a5849bdc240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29aeda7afdee40739eb93a5849bdc240" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33767 } }
I20260925 02:56:51.103592 21734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29aeda7afdee40739eb93a5849bdc240 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:51.105894 21741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29aeda7afdee40739eb93a5849bdc240 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29aeda7afdee40739eb93a5849bdc240. Latest consensus state: current_term: 1 leader_uuid: "29aeda7afdee40739eb93a5849bdc240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29aeda7afdee40739eb93a5849bdc240" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33767 } } }
I20260925 02:56:51.105916 21740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29aeda7afdee40739eb93a5849bdc240 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29aeda7afdee40739eb93a5849bdc240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29aeda7afdee40739eb93a5849bdc240" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33767 } } }
I20260925 02:56:51.106262 21741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29aeda7afdee40739eb93a5849bdc240 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:51.106305 21740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29aeda7afdee40739eb93a5849bdc240 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:51.107173 21746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:51.112494 21746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:51.120644 21746 catalog_manager.cc:1347] Generated new cluster ID: a620a01ad8af424e9fdc078f7b30a298
I20260925 02:56:51.120754 21746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:51.147953 21746 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:51.149456 21746 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:51.157733 21746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29aeda7afdee40739eb93a5849bdc240: Generated new TSK 0
I20260925 02:56:51.158521 21746 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
I20260925 02:56:51.206352 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33767
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:51.524942 21760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:51.525197 21760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:51.525306 21760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:51.532240 21760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:51.532353 21760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:51.532567 21760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 02:56:51.541486 21760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:51.542647 21760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:51.543915 21760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:51.556427 21768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:51.556468 21765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:51.556473 21766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:51.557349 21760 server_base.cc:1034] running on GCE node
I20260925 02:56:51.557966 21760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:51.558892 21760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:51.560122 21760 hybrid_clock.cc:648] HybridClock initialized: now 1790305011560077 us; error 55 us; skew 500 ppm
I20260925 02:56:51.560423 21760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:51 dist-test-slave-8zvk krb5kdc[21653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305011, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:51.563454 21760 init.cc:377] Logged in from keytab as kudu/127.18.196.193@KRBTEST.COM (short username kudu)
I20260925 02:56:51.564720 21760 webserver.cc:460] Webserver started at http://127.18.196.193:41669/ using document root <none> and password file <none>
I20260925 02:56:51.565310 21760 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:51.565402 21760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:51.565637 21760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:51.568380 21760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "0ddf0574bfa3426da4891fb9009fc7f0"
format_stamp: "Formatted at 2026-09-25 02:56:51 on dist-test-slave-8zvk"
server_key: "5b6c2baad9f1fc07563709ee70738637"
server_key_iv: "cf5974c8d725ed124d8f70f133f8c578"
server_key_version: "encryptionkey@0"
I20260925 02:56:51.569090 21760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "0ddf0574bfa3426da4891fb9009fc7f0"
format_stamp: "Formatted at 2026-09-25 02:56:51 on dist-test-slave-8zvk"
server_key: "5b6c2baad9f1fc07563709ee70738637"
server_key_iv: "cf5974c8d725ed124d8f70f133f8c578"
server_key_version: "encryptionkey@0"
I20260925 02:56:51.573800 21760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:56:51.576923 21775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:51.578127 21760 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 02:56:51.578248 21760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "0ddf0574bfa3426da4891fb9009fc7f0"
format_stamp: "Formatted at 2026-09-25 02:56:51 on dist-test-slave-8zvk"
server_key: "5b6c2baad9f1fc07563709ee70738637"
server_key_iv: "cf5974c8d725ed124d8f70f133f8c578"
server_key_version: "encryptionkey@0"
I20260925 02:56:51.578410 21760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:51.610571 21760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:51.614902 21760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:51.615151 21760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:51.616045 21760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:51.617328 21760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:51.617401 21760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:51.617487 21760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:51.617530 21760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:51.649370 21760 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:44579
I20260925 02:56:51.649452 21889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:44579 every 8 connection(s)
I20260925 02:56:51.650612 21760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 02:56:51.654671 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 21760
I20260925 02:56:51.654919 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 02:56:51.655537 19219 external_mini_cluster.cc:1428] Setting key 71460180f3dbd62d7c1d23c45a59ac1d
Sep 25 02:56:51 dist-test-slave-8zvk krb5kdc[21653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305011, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:51.672571 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:51.653154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:59259 (local address 127.18.196.254:33767)
0925 02:56:51.653613 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:51.653619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:51.654584 (+ 965us) server_negotiation.cc:407] Connection header received
0925 02:56:51.656546 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:51.656552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:51.656638 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:51.656803 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:51.658546 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:51.659310 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:51.660879 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:51.661273 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:51.661419 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:51.665394 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:51.665402 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:51.665407 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:51.665462 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:51.669115 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:51.670060 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:51.670064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:51.670066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:51.670177 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:51.670695 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:51.670699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:51.670701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:51.671081 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:51.671205 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:51.672147 (+ 942us) server_negotiation.cc:299] Negotiation successful
0925 02:56:51.672320 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":334,"thread_start_us":216,"threads_started":1}
I20260925 02:56:51.674068 21894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:51.653692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:33767 (local address 127.18.196.193:59259)
0925 02:56:51.654334 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:51.654375 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:51.656266 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:51.657064 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:51.657076 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:51.657728 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:51.658376 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:51.658394 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:51.659473 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:51.659477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:51.660723 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:51.660735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:51.661516 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:51.661524 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:51.661743 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:51.662442 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:51.662471 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:51.665136 (+ 2665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:51.669251 (+ 4115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:51.669263 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:51.669281 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:51.669934 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:51.670290 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:51.670298 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:51.670305 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:51.670547 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:51.671222 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:51.671232 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:51.671722 (+ 490us) client_negotiation.cc:769] Sending connection context
0925 02:56:51.672058 (+ 336us) client_negotiation.cc:240] Negotiation successful
0925 02:56:51.672614 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":254,"threads_started":1}
WARNING: no policy specified for kudu/127.18.196.194@KRBTEST.COM; defaulting to no policy
I20260925 02:56:51.675664 21890 heartbeater.cc:347] Connected to a master server at 127.18.196.254:33767
I20260925 02:56:51.675913 21890 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:51.676707 21890 heartbeater.cc:511] Master 127.18.196.254:33767 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.194@KRBTEST.COM" created.
I20260925 02:56:51.679062 21698 ts_manager.cc:194] Registered new tserver with Master: 0ddf0574bfa3426da4891fb9009fc7f0 (127.18.196.193:44579)
I20260925 02:56:51.681195 21698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.193@KRBTEST.COM'} at 127.18.196.193:59259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
I20260925 02:56:51.702092 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33767
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:52.033403 21897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:52.033679 21897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:52.033840 21897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:52.040763 21897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:52.040853 21897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:52.041054 21897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 02:56:52.050612 21897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:52.051842 21897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:52.053126 21897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:52.066501 21903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:52.066447 21902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:52.066972 21905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:52.067261 21897 server_base.cc:1034] running on GCE node
I20260925 02:56:52.067968 21897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:52.069120 21897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:52.070349 21897 hybrid_clock.cc:648] HybridClock initialized: now 1790305012070282 us; error 50 us; skew 500 ppm
I20260925 02:56:52.070688 21897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:52 dist-test-slave-8zvk krb5kdc[21653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305012, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:52.074288 21897 init.cc:377] Logged in from keytab as kudu/127.18.196.194@KRBTEST.COM (short username kudu)
I20260925 02:56:52.075790 21897 webserver.cc:460] Webserver started at http://127.18.196.194:35273/ using document root <none> and password file <none>
I20260925 02:56:52.076581 21897 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:52.076730 21897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:52.077034 21897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:52.080152 21897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "d1fccdf9597f4ac6a3e6c6e5ef915a0b"
format_stamp: "Formatted at 2026-09-25 02:56:52 on dist-test-slave-8zvk"
server_key: "0a360d5f9bd67a8d97876bc00f8bb9d8"
server_key_iv: "690609aa24365a432fe51b6d0112196a"
server_key_version: "encryptionkey@0"
I20260925 02:56:52.080955 21897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "d1fccdf9597f4ac6a3e6c6e5ef915a0b"
format_stamp: "Formatted at 2026-09-25 02:56:52 on dist-test-slave-8zvk"
server_key: "0a360d5f9bd67a8d97876bc00f8bb9d8"
server_key_iv: "690609aa24365a432fe51b6d0112196a"
server_key_version: "encryptionkey@0"
I20260925 02:56:52.086331 21897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 02:56:52.090052 21912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:52.091457 21897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:56:52.091645 21897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "d1fccdf9597f4ac6a3e6c6e5ef915a0b"
format_stamp: "Formatted at 2026-09-25 02:56:52 on dist-test-slave-8zvk"
server_key: "0a360d5f9bd67a8d97876bc00f8bb9d8"
server_key_iv: "690609aa24365a432fe51b6d0112196a"
server_key_version: "encryptionkey@0"
I20260925 02:56:52.091853 21897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:52.118117 21897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:52.122684 21897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:52.123023 21897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:52.124032 21897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:52.125415 21897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:52.125530 21897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:52.125680 21897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:52.125752 21897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:52.158285 21897 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:44609
I20260925 02:56:52.158327 22026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:44609 every 8 connection(s)
I20260925 02:56:52.159584 21897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 02:56:52.169492 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 21897
I20260925 02:56:52.169698 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 02:56:52.170109 19219 external_mini_cluster.cc:1428] Setting key 201c2775b1fc50a7bdad41ea25a193f2
Sep 25 02:56:52 dist-test-slave-8zvk krb5kdc[21653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305012, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:52.178686 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:52.162072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:56103 (local address 127.18.196.254:33767)
0925 02:56:52.162251 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:52.162256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:52.163376 (+ 1120us) server_negotiation.cc:407] Connection header received
0925 02:56:52.165112 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:52.165116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:52.165182 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:52.165269 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:52.166754 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:52.167524 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:52.169100 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:52.169488 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:52.169640 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:52.173016 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:52.173029 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:52.173033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:52.173064 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:52.175832 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:52.176553 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:52.176558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:52.176563 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:52.176678 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:52.177090 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:52.177097 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:52.177101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:52.177339 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:52.177495 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:52.178131 (+ 636us) server_negotiation.cc:299] Negotiation successful
0925 02:56:52.178342 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:56:52.179607 22029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:52.162502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:33767 (local address 127.18.196.194:56103)
0925 02:56:52.163224 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:52.163266 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:52.164841 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:52.165503 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:52.165511 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:52.165993 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:52.166597 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:52.166616 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:52.167653 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:52.167657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:52.168894 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:52.168909 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:52.169721 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:52.169727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:52.169939 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:52.170629 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:52.170657 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:52.172780 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:52.175989 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:52.175998 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:52.176012 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:52.176417 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:52.176792 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:52.176796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:52.176800 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:52.176971 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:52.177461 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:52.177469 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:52.177776 (+ 307us) client_negotiation.cc:769] Sending connection context
0925 02:56:52.178054 (+ 278us) client_negotiation.cc:240] Negotiation successful
0925 02:56:52.178544 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":318,"threads_started":1}
I20260925 02:56:52.181005 22027 heartbeater.cc:347] Connected to a master server at 127.18.196.254:33767
I20260925 02:56:52.181324 22027 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:52.182153 22027 heartbeater.cc:511] Master 127.18.196.254:33767 requested a full tablet report, sending...
I20260925 02:56:52.183760 21698 ts_manager.cc:194] Registered new tserver with Master: d1fccdf9597f4ac6a3e6c6e5ef915a0b (127.18.196.194:44609)
I20260925 02:56:52.185021 21698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.194@KRBTEST.COM'} at 127.18.196.194:56103
WARNING: no policy specified for kudu/127.18.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
I20260925 02:56:52.214408 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33767
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:52.514400 22032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:52.514643 22032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:52.514729 22032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:52.520946 22032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:52.521025 22032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:52.521127 22032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 02:56:52.529465 22032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:52.530510 22032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:52.531630 22032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:52.543313 22037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:52.543355 22038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:52.543670 22040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:52.544134 22032 server_base.cc:1034] running on GCE node
I20260925 02:56:52.544584 22032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:52.545344 22032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:52.546528 22032 hybrid_clock.cc:648] HybridClock initialized: now 1790305012546495 us; error 27 us; skew 500 ppm
I20260925 02:56:52.546813 22032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:52 dist-test-slave-8zvk krb5kdc[21653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305012, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:52.549643 22032 init.cc:377] Logged in from keytab as kudu/127.18.196.195@KRBTEST.COM (short username kudu)
I20260925 02:56:52.550997 22032 webserver.cc:460] Webserver started at http://127.18.196.195:38073/ using document root <none> and password file <none>
I20260925 02:56:52.551620 22032 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:52.551764 22032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:52.552035 22032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:52.554625 22032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "4338132e110c4543b55f8050262459e7"
format_stamp: "Formatted at 2026-09-25 02:56:52 on dist-test-slave-8zvk"
server_key: "e7ce8090fe7557e842a04626ef62fb05"
server_key_iv: "cb2f69a04d25d8aa3c272fa40d0051e7"
server_key_version: "encryptionkey@0"
I20260925 02:56:52.555437 22032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "4338132e110c4543b55f8050262459e7"
format_stamp: "Formatted at 2026-09-25 02:56:52 on dist-test-slave-8zvk"
server_key: "e7ce8090fe7557e842a04626ef62fb05"
server_key_iv: "cb2f69a04d25d8aa3c272fa40d0051e7"
server_key_version: "encryptionkey@0"
I20260925 02:56:52.559846 22032 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:56:52.563067 22047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:52.564216 22032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:56:52.564369 22032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "4338132e110c4543b55f8050262459e7"
format_stamp: "Formatted at 2026-09-25 02:56:52 on dist-test-slave-8zvk"
server_key: "e7ce8090fe7557e842a04626ef62fb05"
server_key_iv: "cb2f69a04d25d8aa3c272fa40d0051e7"
server_key_version: "encryptionkey@0"
I20260925 02:56:52.564541 22032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:52.583336 22032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:52.587396 22032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:52.587657 22032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:52.588459 22032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:52.589645 22032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:52.589736 22032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:52.589864 22032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:52.589926 22032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:52.621027 22032 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:37967
I20260925 02:56:52.621105 22160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:37967 every 8 connection(s)
I20260925 02:56:52.622270 22032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 02:56:52.631572 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 22032
I20260925 02:56:52.631762 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 02:56:52.632171 19219 external_mini_cluster.cc:1428] Setting key cde4aabad45f7dc2688a6c0cc548d12f
Sep 25 02:56:52 dist-test-slave-8zvk krb5kdc[21653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305012, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:52.640885 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:52.624698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:45449 (local address 127.18.196.254:33767)
0925 02:56:52.624884 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:52.624888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:52.625920 (+ 1032us) server_negotiation.cc:407] Connection header received
0925 02:56:52.627749 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:52.627755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:52.627845 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:52.627953 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:52.629474 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:52.630295 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:52.631871 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:52.632297 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:52.632435 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:52.635869 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:52.635874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:52.635877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:52.635906 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:52.638392 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:52.639069 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:52.639073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:52.639075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:52.639196 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:52.639549 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:52.639552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:52.639554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:52.639801 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:52.639904 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:52.640487 (+ 583us) server_negotiation.cc:299] Negotiation successful
0925 02:56:52.640642 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:56:52.641795 22164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:52.625109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:33767 (local address 127.18.196.195:45449)
0925 02:56:52.625760 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:52.625801 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:52.627475 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:52.628184 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:52.628192 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:52.628661 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:52.629283 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:52.629301 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:52.630450 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:52.630453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:52.631711 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:52.631726 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:52.632535 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:52.632543 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:52.632774 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:52.633441 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:52.633468 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:52.635707 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:52.638510 (+ 2803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:52.638516 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:52.638526 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:52.638967 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:52.639287 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:52.639289 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:52.639291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:52.639456 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:52.639898 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:52.639903 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:52.640173 (+ 270us) client_negotiation.cc:769] Sending connection context
0925 02:56:52.640372 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 02:56:52.640751 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":250,"threads_started":1}
I20260925 02:56:52.643137 22161 heartbeater.cc:347] Connected to a master server at 127.18.196.254:33767
I20260925 02:56:52.643414 22161 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:52.644085 22161 heartbeater.cc:511] Master 127.18.196.254:33767 requested a full tablet report, sending...
I20260925 02:56:52.645491 21698 ts_manager.cc:194] Registered new tserver with Master: 4338132e110c4543b55f8050262459e7 (127.18.196.195:37967)
I20260925 02:56:52.646440 21698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.195@KRBTEST.COM'} at 127.18.196.195:45449
I20260925 02:56:52.647388 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 02:56:52.679586 21892 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:52.677176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57836 (local address 127.18.196.254:33767)
0925 02:56:52.677385 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:52.677390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:52.677882 (+ 492us) server_negotiation.cc:407] Connection header received
0925 02:56:52.678045 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:52.678050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:52.678113 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:52.678204 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:52.679349 (+ 1145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57836: BlockingRecv error: recv got EOF from 127.0.0.1:57836 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:56:52.682879 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 21760
I20260925 02:56:52.684981 21890 heartbeater.cc:503] Master 127.18.196.254:33767 was elected leader, sending a full tablet report...
W20260925 02:56:52.685596 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 21760: Not authorized: Client connection negotiation failed: client connection to 127.18.196.193:44579: server requires authentication, but client does not have Kerberos credentials available
I20260925 02:56:52.685766 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 21760
W20260925 02:56:52.701965 21682 connection.cc:535] server connection from 127.18.196.193:59259 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:56:52.702324 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 21897
W20260925 02:56:52.705053 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 21897: Not authorized: Client connection negotiation failed: client connection to 127.18.196.194:44609: server requires authentication, but client does not have Kerberos credentials available
I20260925 02:56:52.705229 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 21897
I20260925 02:56:52.721779 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 22032
W20260925 02:56:52.724403 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 22032: Not authorized: Client connection negotiation failed: client connection to 127.18.196.195:37967: server requires authentication, but client does not have Kerberos credentials available
I20260925 02:56:52.724552 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 22032
I20260925 02:56:52.739732 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 21666
I20260925 02:56:52.740824 21729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:56:52.841077 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 21666
2026-09-25T02:56:52Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3453 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:54 dist-test-slave-8zvk krb5kdc[22179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:54 dist-test-slave-8zvk krb5kdc[22179](info): set up 2 sockets
Sep 25 02:56:54 dist-test-slave-8zvk krb5kdc[22179](info): commencing operation
krb5kdc: starting...
W20260925 02:56:55.422395 19219 mini_kdc.cc:121] Time spent starting KDC: real 2.538s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 02:56:55 dist-test-slave-8zvk krb5kdc[22179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 02:56:55.591840 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:36279
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:36677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:36279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:55.887686 22192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:55.887933 22192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:55.888015 22192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:55.894512 22192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:55.894599 22192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:55.894642 22192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:55.894696 22192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:55.894734 22192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:55.903873 22192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:36677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:36279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:36279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:55.904961 22192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:55.906085 22192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:55.917464 22198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:55.917527 22201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:55.917552 22199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:55.918251 22192 server_base.cc:1034] running on GCE node
I20260925 02:56:55.919049 22192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:55.920351 22192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:55.921543 22192 hybrid_clock.cc:648] HybridClock initialized: now 1790305015921512 us; error 32 us; skew 500 ppm
I20260925 02:56:55.921898 22192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:55 dist-test-slave-8zvk krb5kdc[22179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305015, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:55.930004 22192 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 02:56:55.931425 22192 webserver.cc:460] Webserver started at http://127.18.196.254:33763/ using document root <none> and password file <none>
I20260925 02:56:55.932082 22192 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:55.932205 22192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:55.932524 22192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:55.935480 22192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "2448df181062441f8aa7e8ed355908b5"
format_stamp: "Formatted at 2026-09-25 02:56:55 on dist-test-slave-8zvk"
server_key: "94234847f40431d89823290f22dd0d3e"
server_key_iv: "c4a298439c6c4862af3c6911df48dccb"
server_key_version: "encryptionkey@0"
I20260925 02:56:55.936228 22192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "2448df181062441f8aa7e8ed355908b5"
format_stamp: "Formatted at 2026-09-25 02:56:55 on dist-test-slave-8zvk"
server_key: "94234847f40431d89823290f22dd0d3e"
server_key_iv: "c4a298439c6c4862af3c6911df48dccb"
server_key_version: "encryptionkey@0"
I20260925 02:56:55.941026 22192 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:56:55.944288 22208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:55.945467 22192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:56:55.945652 22192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "2448df181062441f8aa7e8ed355908b5"
format_stamp: "Formatted at 2026-09-25 02:56:55 on dist-test-slave-8zvk"
server_key: "94234847f40431d89823290f22dd0d3e"
server_key_iv: "c4a298439c6c4862af3c6911df48dccb"
server_key_version: "encryptionkey@0"
I20260925 02:56:55.945829 22192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:55.964567 22192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:55.968259 22192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:55.968532 22192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:55.986033 22192 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:36279
I20260925 02:56:55.986114 22260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:36279 every 8 connection(s)
I20260925 02:56:55.987284 22192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:55.990563 22261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:55.996285 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 22192
I20260925 02:56:55.996518 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:56:55.997342 19219 external_mini_cluster.cc:1428] Setting key be09626dde2e1bf2b209032508f72714
I20260925 02:56:56.001199 22261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2448df181062441f8aa7e8ed355908b5: Bootstrap starting.
I20260925 02:56:56.004781 22261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2448df181062441f8aa7e8ed355908b5: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:56.005817 22261 log.cc:824] T 00000000000000000000000000000000 P 2448df181062441f8aa7e8ed355908b5: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:56.008594 22261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2448df181062441f8aa7e8ed355908b5: No bootstrap required, opened a new log
Sep 25 02:56:56 dist-test-slave-8zvk krb5kdc[22179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:56.016389 22261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2448df181062441f8aa7e8ed355908b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2448df181062441f8aa7e8ed355908b5" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36279 } }
I20260925 02:56:56.016755 22261 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2448df181062441f8aa7e8ed355908b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:56.016885 22261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2448df181062441f8aa7e8ed355908b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2448df181062441f8aa7e8ed355908b5, State: Initialized, Role: FOLLOWER
I20260925 02:56:56.017488 22261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2448df181062441f8aa7e8ed355908b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2448df181062441f8aa7e8ed355908b5" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36279 } }
I20260925 02:56:56.017776 22261 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2448df181062441f8aa7e8ed355908b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:56.017940 22261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2448df181062441f8aa7e8ed355908b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:56.018090 22261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2448df181062441f8aa7e8ed355908b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:56.019362 22264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:55.999216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35416 (local address 127.18.196.254:36279)
0925 02:56:55.999973 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:55.999983 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:56.000018 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:56:56.001618 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:56.001640 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:56.001985 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:56.002263 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:56.004203 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:56.005474 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:56.006996 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:56.007571 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:56.007714 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:56.012120 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:56.012126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:56.012142 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:56.012175 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:56.015233 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:56.015928 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:56.015934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:56.015939 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:56.016093 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:56.016468 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:56.016474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:56.016481 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:56.017132 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:56.017416 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:56.017858 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 02:56:56.018395 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":369,"threads_started":1}
I20260925 02:56:56.020291 22261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2448df181062441f8aa7e8ed355908b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2448df181062441f8aa7e8ed355908b5" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36279 } }
I20260925 02:56:56.020782 22261 leader_election.cc:302] T 00000000000000000000000000000000 P 2448df181062441f8aa7e8ed355908b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2448df181062441f8aa7e8ed355908b5; no voters:
I20260925 02:56:56.021467 22261 leader_election.cc:288] T 00000000000000000000000000000000 P 2448df181062441f8aa7e8ed355908b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:56.021566 22266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2448df181062441f8aa7e8ed355908b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:56.022154 22266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2448df181062441f8aa7e8ed355908b5 [term 1 LEADER]: Becoming Leader. State: Replica: 2448df181062441f8aa7e8ed355908b5, State: Running, Role: LEADER
I20260925 02:56:56.022965 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:56:56.022804 22266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2448df181062441f8aa7e8ed355908b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2448df181062441f8aa7e8ed355908b5" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36279 } }
I20260925 02:56:56.024186 22261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2448df181062441f8aa7e8ed355908b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:56.025362 22268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2448df181062441f8aa7e8ed355908b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2448df181062441f8aa7e8ed355908b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2448df181062441f8aa7e8ed355908b5" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36279 } } }
I20260925 02:56:56.025877 22268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2448df181062441f8aa7e8ed355908b5 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:56.025723 22267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2448df181062441f8aa7e8ed355908b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2448df181062441f8aa7e8ed355908b5. Latest consensus state: current_term: 1 leader_uuid: "2448df181062441f8aa7e8ed355908b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2448df181062441f8aa7e8ed355908b5" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36279 } } }
I20260925 02:56:56.025972 22267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2448df181062441f8aa7e8ed355908b5 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:56.028061 22270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:56.033388 22270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:56.048018 22270 catalog_manager.cc:1347] Generated new cluster ID: 1e4c5ecb006a49fab3e83c831d7c1fd5
I20260925 02:56:56.048103 22270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:56.061497 22270 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:56.063519 22270 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:56.074035 22270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2448df181062441f8aa7e8ed355908b5: Generated new TSK 0
I20260925 02:56:56.074967 22270 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:56:56.076619 22264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:56.067905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35426 (local address 127.18.196.254:36279)
0925 02:56:56.068098 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:56.068104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:56.068643 (+ 539us) server_negotiation.cc:407] Connection header received
0925 02:56:56.068819 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:56.068824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:56.068912 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:56.069011 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:56.070264 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:56.071453 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:56.073161 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:56.073675 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:56.073815 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:56.075108 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:56.075113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:56.075117 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:56.075871 (+ 754us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:56.075954 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:56.076004 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:56.076111 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:56.076254 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 02:56:56.076322 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":73}
I20260925 02:56:56.083617 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 22192
I20260925 02:56:56.084738 22255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:56:56.170866 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 22192
2026-09-25T02:56:56Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (3319 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:56 dist-test-slave-8zvk krb5kdc[22296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:56 dist-test-slave-8zvk krb5kdc[22296](info): set up 2 sockets
Sep 25 02:56:56 dist-test-slave-8zvk krb5kdc[22296](info): commencing operation
krb5kdc: starting...
W20260925 02:56:57.238396 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 02:56:57 dist-test-slave-8zvk krb5kdc[22296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 02:56:57.414889 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:46295
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:32919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:46295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:57.713027 22309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:57.713261 22309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:57.713347 22309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:57.720247 22309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:57.720337 22309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:57.720386 22309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:57.720463 22309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:57.720546 22309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:57.729679 22309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:32919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:46295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:46295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:57.730767 22309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:57.731904 22309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:57.742717 22315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:57.742770 22314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:57.742753 22317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:57.743464 22309 server_base.cc:1034] running on GCE node
I20260925 02:56:57.744199 22309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:57.745442 22309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:57.746650 22309 hybrid_clock.cc:648] HybridClock initialized: now 1790305017746612 us; error 45 us; skew 500 ppm
I20260925 02:56:57.746985 22309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:57 dist-test-slave-8zvk krb5kdc[22296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305017, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:57.749657 22309 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 02:56:57.750944 22309 webserver.cc:460] Webserver started at http://127.18.196.254:46207/ using document root <none> and password file <none>
I20260925 02:56:57.751518 22309 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:57.751657 22309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:57.751943 22309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:57.754609 22309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "a2baaa7542114e9e8c0a0b273d1905a2"
format_stamp: "Formatted at 2026-09-25 02:56:57 on dist-test-slave-8zvk"
server_key: "e49e6552226156d61cb92495a5a904ea"
server_key_iv: "c93766182637ae1d4bd16e2f86f09145"
server_key_version: "encryptionkey@0"
I20260925 02:56:57.755383 22309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "a2baaa7542114e9e8c0a0b273d1905a2"
format_stamp: "Formatted at 2026-09-25 02:56:57 on dist-test-slave-8zvk"
server_key: "e49e6552226156d61cb92495a5a904ea"
server_key_iv: "c93766182637ae1d4bd16e2f86f09145"
server_key_version: "encryptionkey@0"
I20260925 02:56:57.760098 22309 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 02:56:57.763240 22324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:57.764424 22309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 02:56:57.764600 22309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "a2baaa7542114e9e8c0a0b273d1905a2"
format_stamp: "Formatted at 2026-09-25 02:56:57 on dist-test-slave-8zvk"
server_key: "e49e6552226156d61cb92495a5a904ea"
server_key_iv: "c93766182637ae1d4bd16e2f86f09145"
server_key_version: "encryptionkey@0"
I20260925 02:56:57.764797 22309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:57.785140 22309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:57.788628 22309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:57.788918 22309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:57.806620 22309 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:46295
I20260925 02:56:57.806658 22376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:46295 every 8 connection(s)
I20260925 02:56:57.807889 22309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:57.809839 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 22309
I20260925 02:56:57.810086 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:56:57.810941 19219 external_mini_cluster.cc:1428] Setting key ceb44f78084b7cfc36930ebf8f832ec0
I20260925 02:56:57.812999 22377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 02:56:57 dist-test-slave-8zvk krb5kdc[22296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:57.828056 22377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2baaa7542114e9e8c0a0b273d1905a2: Bootstrap starting.
I20260925 02:56:57.831457 22377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2baaa7542114e9e8c0a0b273d1905a2: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:57.832429 22377 log.cc:824] T 00000000000000000000000000000000 P a2baaa7542114e9e8c0a0b273d1905a2: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:57.834962 22380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:57.812956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.18.196.254:46295)
0925 02:56:57.813988 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:57.814000 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:57.814040 (+ 40us) server_negotiation.cc:407] Connection header received
0925 02:56:57.815739 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:57.815761 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:57.816125 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:57.816438 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:57.818306 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:57.820111 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:57.821974 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:57.822615 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:57.822761 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:57.827624 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:57.827631 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:57.827645 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:57.827682 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:57.830565 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:57.831329 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:57.831335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:57.831340 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:57.831542 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:57.831925 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:57.831931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:57.831935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:57.832445 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:57.832757 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:57.833250 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 02:56:57.833656 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":552,"threads_started":1}
I20260925 02:56:57.835677 22377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2baaa7542114e9e8c0a0b273d1905a2: No bootstrap required, opened a new log
I20260925 02:56:57.842676 22377 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2baaa7542114e9e8c0a0b273d1905a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2baaa7542114e9e8c0a0b273d1905a2" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 46295 } }
I20260925 02:56:57.843075 22377 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2baaa7542114e9e8c0a0b273d1905a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:57.843168 22377 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2baaa7542114e9e8c0a0b273d1905a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2baaa7542114e9e8c0a0b273d1905a2, State: Initialized, Role: FOLLOWER
I20260925 02:56:57.843779 22377 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2baaa7542114e9e8c0a0b273d1905a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2baaa7542114e9e8c0a0b273d1905a2" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 46295 } }
I20260925 02:56:57.844071 22377 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2baaa7542114e9e8c0a0b273d1905a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:57.844218 22377 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2baaa7542114e9e8c0a0b273d1905a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:57.844381 22377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2baaa7542114e9e8c0a0b273d1905a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:57.846446 22377 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2baaa7542114e9e8c0a0b273d1905a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2baaa7542114e9e8c0a0b273d1905a2" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 46295 } }
I20260925 02:56:57.847020 22377 leader_election.cc:302] T 00000000000000000000000000000000 P a2baaa7542114e9e8c0a0b273d1905a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2baaa7542114e9e8c0a0b273d1905a2; no voters:
I20260925 02:56:57.847591 22377 leader_election.cc:288] T 00000000000000000000000000000000 P a2baaa7542114e9e8c0a0b273d1905a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:57.847719 22382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2baaa7542114e9e8c0a0b273d1905a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:57.848387 22382 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2baaa7542114e9e8c0a0b273d1905a2 [term 1 LEADER]: Becoming Leader. State: Replica: a2baaa7542114e9e8c0a0b273d1905a2, State: Running, Role: LEADER
I20260925 02:56:57.848881 22382 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2baaa7542114e9e8c0a0b273d1905a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2baaa7542114e9e8c0a0b273d1905a2" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 46295 } }
I20260925 02:56:57.849406 22377 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2baaa7542114e9e8c0a0b273d1905a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:57.851338 22383 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2baaa7542114e9e8c0a0b273d1905a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2baaa7542114e9e8c0a0b273d1905a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2baaa7542114e9e8c0a0b273d1905a2" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 46295 } } }
I20260925 02:56:57.851686 22383 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2baaa7542114e9e8c0a0b273d1905a2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:57.853267 22388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:57.853258 22384 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2baaa7542114e9e8c0a0b273d1905a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2baaa7542114e9e8c0a0b273d1905a2. Latest consensus state: current_term: 1 leader_uuid: "a2baaa7542114e9e8c0a0b273d1905a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2baaa7542114e9e8c0a0b273d1905a2" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 46295 } } }
I20260925 02:56:57.854008 22384 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2baaa7542114e9e8c0a0b273d1905a2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:57.858345 22388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:57.867244 22388 catalog_manager.cc:1347] Generated new cluster ID: e18508d48afb4a3cbed046725d65ab18
I20260925 02:56:57.867369 22388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:57.889715 22388 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:57.891016 22388 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:57.898155 22388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2baaa7542114e9e8c0a0b273d1905a2: Generated new TSK 0
I20260925 02:56:57.898797 22388 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
I20260925 02:56:57.955055 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:46295
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:32919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:58.280910 22403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:58.281172 22403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:58.281284 22403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:58.288226 22403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:58.288338 22403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:58.288446 22403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 02:56:58.297731 22403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:32919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:46295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:58.298978 22403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:58.300213 22403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:58.312980 22408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:58.312927 22411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:58.313050 22409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:58.314087 22403 server_base.cc:1034] running on GCE node
I20260925 02:56:58.314630 22403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:58.315559 22403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:58.316742 22403 hybrid_clock.cc:648] HybridClock initialized: now 1790305018316709 us; error 28 us; skew 500 ppm
I20260925 02:56:58.317020 22403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:58 dist-test-slave-8zvk krb5kdc[22296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305018, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:58.319742 22403 init.cc:377] Logged in from keytab as kudu/127.18.196.193@KRBTEST.COM (short username kudu)
I20260925 02:56:58.320986 22403 webserver.cc:460] Webserver started at http://127.18.196.193:37145/ using document root <none> and password file <none>
I20260925 02:56:58.321636 22403 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:58.321759 22403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:58.322043 22403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:58.324791 22403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "380a9eb769db4d239bce36ada387768e"
format_stamp: "Formatted at 2026-09-25 02:56:58 on dist-test-slave-8zvk"
server_key: "bf196e4c1df684617b8757cc3cd1f9fd"
server_key_iv: "b1647cce222d22004e3534636fa9f019"
server_key_version: "encryptionkey@0"
I20260925 02:56:58.325549 22403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "380a9eb769db4d239bce36ada387768e"
format_stamp: "Formatted at 2026-09-25 02:56:58 on dist-test-slave-8zvk"
server_key: "bf196e4c1df684617b8757cc3cd1f9fd"
server_key_iv: "b1647cce222d22004e3534636fa9f019"
server_key_version: "encryptionkey@0"
I20260925 02:56:58.330397 22403 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260925 02:56:58.333592 22418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:58.335031 22403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:56:58.335187 22403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "380a9eb769db4d239bce36ada387768e"
format_stamp: "Formatted at 2026-09-25 02:56:58 on dist-test-slave-8zvk"
server_key: "bf196e4c1df684617b8757cc3cd1f9fd"
server_key_iv: "b1647cce222d22004e3534636fa9f019"
server_key_version: "encryptionkey@0"
I20260925 02:56:58.335381 22403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:58.356986 22403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:58.360394 22403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:58.360662 22403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:58.361555 22403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:58.362735 22403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:58.362825 22403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:58.362972 22403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:58.363047 22403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:58.394529 22403 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:41823
I20260925 02:56:58.394605 22534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:41823 every 8 connection(s)
I20260925 02:56:58.395897 22403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 02:56:58.402594 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 22403
I20260925 02:56:58.402779 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 02:56:58.403471 19219 external_mini_cluster.cc:1428] Setting key 9533446637dcae4b51ad7de616fbd3d7
Sep 25 02:56:58 dist-test-slave-8zvk krb5kdc[22296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305018, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:58.415326 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:58.398408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:46705 (local address 127.18.196.254:46295)
0925 02:56:58.398927 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:58.398932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:58.399723 (+ 791us) server_negotiation.cc:407] Connection header received
0925 02:56:58.401497 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:58.401502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:58.401591 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:58.401711 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:58.403214 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:58.404020 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:58.405496 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:58.405888 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:58.406014 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:58.409537 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:58.409542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:58.409545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:58.409575 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:58.412400 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:58.413102 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:58.413107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:58.413109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:58.413220 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:58.413643 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:58.413648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:58.413656 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:58.413907 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:58.414059 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:58.414712 (+ 653us) server_negotiation.cc:299] Negotiation successful
0925 02:56:58.414969 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":265,"threads_started":1}
I20260925 02:56:58.416277 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:58.398924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46295 (local address 127.18.196.193:46705)
0925 02:56:58.399586 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:58.399621 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:58.401216 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:58.401942 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:58.401956 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:58.402414 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:58.403055 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:58.403073 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:58.404156 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:58.404166 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:58.405342 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:58.405354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:58.406101 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:58.406109 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:58.406326 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:58.407022 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:58.407055 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:58.409326 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:58.412550 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:58.412559 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:58.412573 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:58.412974 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:58.413325 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:58.413330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:58.413334 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:58.413527 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:58.414047 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:58.414053 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:58.414357 (+ 304us) client_negotiation.cc:769] Sending connection context
0925 02:56:58.414608 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 02:56:58.415144 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":252,"threads_started":1}
I20260925 02:56:58.417734 22535 heartbeater.cc:347] Connected to a master server at 127.18.196.254:46295
I20260925 02:56:58.418083 22535 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:58.419018 22535 heartbeater.cc:511] Master 127.18.196.254:46295 requested a full tablet report, sending...
I20260925 02:56:58.421232 22342 ts_manager.cc:194] Registered new tserver with Master: 380a9eb769db4d239bce36ada387768e (127.18.196.193:41823)
WARNING: no policy specified for kudu/127.18.196.194@KRBTEST.COM; defaulting to no policy
I20260925 02:56:58.422740 22342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.193@KRBTEST.COM'} at 127.18.196.193:46705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
I20260925 02:56:58.447259 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:46295
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:32919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:58.763823 22541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:58.764108 22541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:58.764271 22541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:58.771297 22541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:58.771422 22541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:58.771587 22541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 02:56:58.780740 22541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:32919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:46295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:58.781929 22541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:58.783269 22541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:58.795912 22547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:58.795931 22546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:58.795991 22549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:58.796800 22541 server_base.cc:1034] running on GCE node
I20260925 02:56:58.797435 22541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:58.798317 22541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:58.799515 22541 hybrid_clock.cc:648] HybridClock initialized: now 1790305018799462 us; error 50 us; skew 500 ppm
I20260925 02:56:58.799809 22541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:58 dist-test-slave-8zvk krb5kdc[22296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305018, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:58.802706 22541 init.cc:377] Logged in from keytab as kudu/127.18.196.194@KRBTEST.COM (short username kudu)
I20260925 02:56:58.804078 22541 webserver.cc:460] Webserver started at http://127.18.196.194:39593/ using document root <none> and password file <none>
I20260925 02:56:58.804708 22541 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:58.804829 22541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:58.805119 22541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:58.808766 22541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "ee755f03b3d8458a8c7367657000cce1"
format_stamp: "Formatted at 2026-09-25 02:56:58 on dist-test-slave-8zvk"
server_key: "ce647200a47a1e1efc9ceb94fa1ce141"
server_key_iv: "e26e141ff5cf4896b636725a92fa3b8f"
server_key_version: "encryptionkey@0"
I20260925 02:56:58.809667 22541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "ee755f03b3d8458a8c7367657000cce1"
format_stamp: "Formatted at 2026-09-25 02:56:58 on dist-test-slave-8zvk"
server_key: "ce647200a47a1e1efc9ceb94fa1ce141"
server_key_iv: "e26e141ff5cf4896b636725a92fa3b8f"
server_key_version: "encryptionkey@0"
I20260925 02:56:58.815989 22541 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260925 02:56:58.820139 22557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:58.821712 22541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:56:58.821851 22541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "ee755f03b3d8458a8c7367657000cce1"
format_stamp: "Formatted at 2026-09-25 02:56:58 on dist-test-slave-8zvk"
server_key: "ce647200a47a1e1efc9ceb94fa1ce141"
server_key_iv: "e26e141ff5cf4896b636725a92fa3b8f"
server_key_version: "encryptionkey@0"
I20260925 02:56:58.822055 22541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:58.843327 22541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:58.847028 22541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:58.847262 22541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:58.848088 22541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:58.849342 22541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:58.849416 22541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:58.849504 22541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:58.849545 22541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:58.881677 22541 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:33249
I20260925 02:56:58.881719 22673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:33249 every 8 connection(s)
I20260925 02:56:58.883028 22541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 02:56:58.884809 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 22541
I20260925 02:56:58.885011 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 02:56:58.885572 19219 external_mini_cluster.cc:1428] Setting key e44e582a8e503434d6b6c1bed036cb6b
Sep 25 02:56:58 dist-test-slave-8zvk krb5kdc[22296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305018, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:58.905539 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:58.885583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:60337 (local address 127.18.196.254:46295)
0925 02:56:58.885792 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:58.885798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:58.887022 (+ 1224us) server_negotiation.cc:407] Connection header received
0925 02:56:58.889586 (+ 2564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:58.889590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:58.889651 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:58.889730 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:58.891784 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:58.892517 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:58.894668 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:58.895409 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:58.895545 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:58.899492 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:58.899498 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:58.899501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:58.899530 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:58.902302 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:58.903265 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:58.903269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:58.903271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:58.903405 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:58.903921 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:58.903924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:58.903926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:58.904164 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:58.904273 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:58.905135 (+ 862us) server_negotiation.cc:299] Negotiation successful
0925 02:56:58.905301 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
WARNING: no policy specified for kudu/127.18.196.195@KRBTEST.COM; defaulting to no policy
I20260925 02:56:58.907079 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:58.886004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46295 (local address 127.18.196.194:60337)
0925 02:56:58.886892 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:58.886929 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:58.889292 (+ 2363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:58.890009 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:58.890024 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:58.890777 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:58.891600 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:58.891629 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:58.892680 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:58.892687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:58.894529 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:58.894541 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:58.895600 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:58.895605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:58.895799 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:58.896505 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:58.896531 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:58.899284 (+ 2753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:58.902448 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:58.902458 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:58.902474 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:58.903139 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:58.903526 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:58.903534 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:58.903538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:58.903798 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:58.904289 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:58.904299 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:58.904722 (+ 423us) client_negotiation.cc:769] Sending connection context
0925 02:56:58.905045 (+ 323us) client_negotiation.cc:240] Negotiation successful
0925 02:56:58.905583 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":367,"threads_started":1}
I20260925 02:56:58.908419 22675 heartbeater.cc:347] Connected to a master server at 127.18.196.254:46295
I20260925 02:56:58.908715 22675 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.195@KRBTEST.COM" created.
I20260925 02:56:58.909483 22675 heartbeater.cc:511] Master 127.18.196.254:46295 requested a full tablet report, sending...
I20260925 02:56:58.911101 22342 ts_manager.cc:194] Registered new tserver with Master: ee755f03b3d8458a8c7367657000cce1 (127.18.196.194:33249)
I20260925 02:56:58.911907 22342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.194@KRBTEST.COM'} at 127.18.196.194:60337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
I20260925 02:56:58.931692 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:46295
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:32919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:59.245848 22680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:59.246105 22680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:59.246289 22680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:59.253609 22680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:59.253733 22680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:59.253918 22680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 02:56:59.263417 22680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:32919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:46295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:56:59.264613 22680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:59.265813 22680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:59.278478 22688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:59.278517 22685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:59.278431 22686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:59.279394 22680 server_base.cc:1034] running on GCE node
I20260925 02:56:59.280053 22680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:59.280946 22680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:59.282159 22680 hybrid_clock.cc:648] HybridClock initialized: now 1790305019282102 us; error 48 us; skew 500 ppm
I20260925 02:56:59.282450 22680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:56:59 dist-test-slave-8zvk krb5kdc[22296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305019, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:59.285574 22680 init.cc:377] Logged in from keytab as kudu/127.18.196.195@KRBTEST.COM (short username kudu)
I20260925 02:56:59.286954 22680 webserver.cc:460] Webserver started at http://127.18.196.195:43991/ using document root <none> and password file <none>
I20260925 02:56:59.287627 22680 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:59.287751 22680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:59.288045 22680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:59.290782 22680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "40ab223f0b6a44ff8e69f90d85b3d4e7"
format_stamp: "Formatted at 2026-09-25 02:56:59 on dist-test-slave-8zvk"
server_key: "435499c1dd6e19ee85dbedd630b8056e"
server_key_iv: "58d024b58e06e1deefbb76d933d9dbc0"
server_key_version: "encryptionkey@0"
I20260925 02:56:59.291605 22680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "40ab223f0b6a44ff8e69f90d85b3d4e7"
format_stamp: "Formatted at 2026-09-25 02:56:59 on dist-test-slave-8zvk"
server_key: "435499c1dd6e19ee85dbedd630b8056e"
server_key_iv: "58d024b58e06e1deefbb76d933d9dbc0"
server_key_version: "encryptionkey@0"
I20260925 02:56:59.296774 22680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 02:56:59.299966 22696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:59.301155 22680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260925 02:56:59.301338 22680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "40ab223f0b6a44ff8e69f90d85b3d4e7"
format_stamp: "Formatted at 2026-09-25 02:56:59 on dist-test-slave-8zvk"
server_key: "435499c1dd6e19ee85dbedd630b8056e"
server_key_iv: "58d024b58e06e1deefbb76d933d9dbc0"
server_key_version: "encryptionkey@0"
I20260925 02:56:59.301542 22680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:59.314519 22680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:59.318315 22680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:59.318619 22680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:59.319597 22680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:59.320734 22680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:59.320801 22680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:59.320937 22680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:59.321019 22680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:59.352149 22680 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:42241
I20260925 02:56:59.352190 22813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:42241 every 8 connection(s)
I20260925 02:56:59.353305 22680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 02:56:59.358199 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 22680
I20260925 02:56:59.358388 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 02:56:59.358892 19219 external_mini_cluster.cc:1428] Setting key 697eb3ebf74433c4aff1c7fc1a922f44
Sep 25 02:56:59 dist-test-slave-8zvk krb5kdc[22296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305019, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:59.372236 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:59.355821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:59635 (local address 127.18.196.254:46295)
0925 02:56:59.356026 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:59.356032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:59.357064 (+ 1032us) server_negotiation.cc:407] Connection header received
0925 02:56:59.358907 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:59.358912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:59.359067 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:59.359168 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:59.360939 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:59.361900 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:59.363452 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:59.363881 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:59.363984 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:59.367165 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:59.367171 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:59.367174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:59.367203 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:59.369627 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:59.370232 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:59.370236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:59.370238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:59.370373 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:59.370711 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:59.370714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:59.370716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:59.371037 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:59.371160 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:59.371742 (+ 582us) server_negotiation.cc:299] Negotiation successful
0925 02:56:59.371929 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:56:59.373231 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:59.356246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46295 (local address 127.18.196.195:59635)
0925 02:56:59.356906 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:59.356942 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:59.358597 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:59.359402 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:59.359413 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:59.360077 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:59.360773 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:59.360799 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:59.362031 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:59.362035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:59.363304 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:59.363317 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:59.364072 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:59.364077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:59.364258 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:59.364887 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:59.364908 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:59.366982 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:59.369754 (+ 2772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:59.369760 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:59.369770 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:59.370132 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:59.370462 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:59.370465 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:59.370467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:59.370628 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:59.371164 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:59.371170 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:59.371455 (+ 285us) client_negotiation.cc:769] Sending connection context
0925 02:56:59.371663 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 02:56:59.371995 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":259,"threads_started":1}
I20260925 02:56:59.374428 22814 heartbeater.cc:347] Connected to a master server at 127.18.196.254:46295
I20260925 02:56:59.374716 22814 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:59.375396 22814 heartbeater.cc:511] Master 127.18.196.254:46295 requested a full tablet report, sending...
I20260925 02:56:59.376677 22342 ts_manager.cc:194] Registered new tserver with Master: 40ab223f0b6a44ff8e69f90d85b3d4e7 (127.18.196.195:42241)
I20260925 02:56:59.377450 22342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.195@KRBTEST.COM'} at 127.18.196.195:59635
I20260925 02:56:59.385696 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:56:59 dist-test-slave-8zvk krb5kdc[22296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305019, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
I20260925 02:56:59.426079 22535 heartbeater.cc:503] Master 127.18.196.254:46295 was elected leader, sending a full tablet report...
Sep 25 02:56:59 dist-test-slave-8zvk krb5kdc[22296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305019, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:56:59.433269 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:59.418669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60844 (local address 127.18.196.254:46295)
0925 02:56:59.418903 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:59.418911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:59.419323 (+ 412us) server_negotiation.cc:407] Connection header received
0925 02:56:59.419540 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:59.419545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:59.419609 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:59.419709 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:59.421063 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:59.421805 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:59.423121 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:59.423519 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:59.423650 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:59.428292 (+ 4642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:59.428299 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:59.428303 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:59.428347 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:59.430703 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:59.431365 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:59.431370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:59.431374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:59.431508 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:59.431878 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:59.431883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:59.431890 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:59.432198 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:59.432452 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:59.432752 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 02:56:59.432973 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 02:56:59.433784 22342 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60844
I20260925 02:56:59.435443 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 22403
Sep 25 02:56:59 dist-test-slave-8zvk krb5kdc[22296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305019, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.18.196.193@KRBTEST.COM
I20260925 02:56:59.449961 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:59.436369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52278 (local address 127.18.196.193:41823)
0925 02:56:59.436950 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:59.436963 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:59.436987 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:56:59.437042 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:59.437048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:59.437239 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:59.437380 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:59.438480 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:59.439630 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:59.440776 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:59.441298 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:59.441412 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:59.445093 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:59.445103 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:59.445116 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:59.445145 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:59.447689 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:59.448260 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:59.448266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:59.448268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:59.448399 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:59.448688 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:59.448691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:59.448693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:59.449009 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:59.449202 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:59.449419 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 02:56:59.449688 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":304,"threads_started":1}
W20260925 02:56:59.451156 22530 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52278
W20260925 02:56:59.452172 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 22403: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:56:59.452384 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 22403
I20260925 02:56:59.468299 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 22541
Sep 25 02:56:59 dist-test-slave-8zvk krb5kdc[22296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305019, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.18.196.194@KRBTEST.COM
I20260925 02:56:59.482650 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:59.469134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41364 (local address 127.18.196.194:33249)
0925 02:56:59.469631 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:59.469638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:59.469660 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:56:59.469714 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:59.469720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:59.469919 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:59.470044 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:59.471150 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:59.472085 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:59.473172 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:59.473690 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:59.473788 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:59.477565 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:59.477575 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:59.477587 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:59.477616 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:59.480440 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:59.480996 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:59.481002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:59.481004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:59.481132 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:59.481461 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:59.481464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:59.481466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:59.481759 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:59.481993 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:59.482194 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 02:56:59.482374 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":255,"threads_started":1}
W20260925 02:56:59.483789 22669 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41364
W20260925 02:56:59.484736 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 22541: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:56:59.484881 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 22541
I20260925 02:56:59.500785 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 22680
Sep 25 02:56:59 dist-test-slave-8zvk krb5kdc[22296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305019, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.18.196.195@KRBTEST.COM
I20260925 02:56:59.516456 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:59.501866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48818 (local address 127.18.196.195:42241)
0925 02:56:59.502621 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:59.502628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:59.502654 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:56:59.502714 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:59.502720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:59.502978 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:59.503107 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:59.504341 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:59.505430 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:59.506715 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:59.507365 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:59.507471 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:59.511433 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:59.511442 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:59.511454 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:59.511483 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:59.514157 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:59.514722 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:59.514728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:59.514731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:59.514895 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:59.515197 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:59.515201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:59.515202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:59.515508 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:59.515718 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:59.515975 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 02:56:59.516158 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":415,"threads_started":1}
W20260925 02:56:59.517661 22809 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48818
W20260925 02:56:59.518718 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 22680: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:56:59.518898 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 22680
I20260925 02:56:59.534446 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 22309
I20260925 02:56:59.535440 22372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:56:59.631423 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 22309
2026-09-25T02:56:59Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3468 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:59 dist-test-slave-8zvk krb5kdc[22832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:59 dist-test-slave-8zvk krb5kdc[22832](info): set up 2 sockets
Sep 25 02:56:59 dist-test-slave-8zvk krb5kdc[22832](info): commencing operation
krb5kdc: starting...
W20260925 02:57:00.703516 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 02:57:00 dist-test-slave-8zvk krb5kdc[22832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:57:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:57:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 02:57:00.888365 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:33103
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:42171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:33103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:01.190332 22845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:01.190631 22845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:01.190771 22845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:01.198052 22845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:01.198182 22845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:01.198287 22845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:01.198383 22845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:01.198467 22845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:01.208108 22845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:42171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:33103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:33103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:01.209300 22845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:01.210544 22845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:01.224113 22851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:01.224206 22853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:01.224248 22850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:01.224992 22845 server_base.cc:1034] running on GCE node
I20260925 02:57:01.225721 22845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:01.227035 22845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:01.228240 22845 hybrid_clock.cc:648] HybridClock initialized: now 1790305021228204 us; error 26 us; skew 500 ppm
I20260925 02:57:01.228606 22845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:01 dist-test-slave-8zvk krb5kdc[22832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305021, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:01.237331 22845 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 02:57:01.238916 22845 webserver.cc:460] Webserver started at http://127.18.196.254:38639/ using document root <none> and password file <none>
I20260925 02:57:01.239605 22845 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:01.239722 22845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:01.240007 22845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:01.242959 22845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "2d192ed873304f7698875e605d417939"
format_stamp: "Formatted at 2026-09-25 02:57:01 on dist-test-slave-8zvk"
server_key: "24eb11dfd95be717e541af2762cfec67"
server_key_iv: "5cee2071d48c01181ccc7cc9ce9925b0"
server_key_version: "encryptionkey@0"
I20260925 02:57:01.243752 22845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "2d192ed873304f7698875e605d417939"
format_stamp: "Formatted at 2026-09-25 02:57:01 on dist-test-slave-8zvk"
server_key: "24eb11dfd95be717e541af2762cfec67"
server_key_iv: "5cee2071d48c01181ccc7cc9ce9925b0"
server_key_version: "encryptionkey@0"
I20260925 02:57:01.248620 22845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260925 02:57:01.251860 22860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:01.253023 22845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:57:01.253183 22845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "2d192ed873304f7698875e605d417939"
format_stamp: "Formatted at 2026-09-25 02:57:01 on dist-test-slave-8zvk"
server_key: "24eb11dfd95be717e541af2762cfec67"
server_key_iv: "5cee2071d48c01181ccc7cc9ce9925b0"
server_key_version: "encryptionkey@0"
I20260925 02:57:01.253369 22845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:01.267828 22845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:01.271898 22845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:01.272150 22845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:01.289880 22845 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:33103
I20260925 02:57:01.289909 22911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:33103 every 8 connection(s)
I20260925 02:57:01.291213 22845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:57:01.292219 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 22845
I20260925 02:57:01.292457 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:57:01.293251 19219 external_mini_cluster.cc:1428] Setting key 0ec13bf5f371cd3dcf6b850d48e5c64d
I20260925 02:57:01.295841 22912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 02:57:01 dist-test-slave-8zvk krb5kdc[22832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:57:01.308722 22912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d192ed873304f7698875e605d417939: Bootstrap starting.
I20260925 02:57:01.312898 22912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d192ed873304f7698875e605d417939: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:01.314100 22912 log.cc:824] T 00000000000000000000000000000000 P 2d192ed873304f7698875e605d417939: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:01.317493 22912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d192ed873304f7698875e605d417939: No bootstrap required, opened a new log
I20260925 02:57:01.317526 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:01.295376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40562 (local address 127.18.196.254:33103)
0925 02:57:01.296189 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:01.296200 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:01.296239 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:57:01.297813 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:01.297834 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:01.298204 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:01.298533 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:01.300341 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:01.301672 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:01.303441 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:01.304086 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:01.304222 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:01.308907 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:01.308918 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:01.308945 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:01.309009 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:01.312904 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:01.313624 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:01.313632 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:01.313658 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:01.313881 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:01.314281 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:01.314285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:01.314288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:01.315032 (+ 744us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:01.315296 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:01.315812 (+ 516us) server_negotiation.cc:299] Negotiation successful
0925 02:57:01.316243 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":370,"threads_started":1}
I20260925 02:57:01.325278 22912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d192ed873304f7698875e605d417939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d192ed873304f7698875e605d417939" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33103 } }
I20260925 02:57:01.325601 22912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d192ed873304f7698875e605d417939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:01.325700 22912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d192ed873304f7698875e605d417939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d192ed873304f7698875e605d417939, State: Initialized, Role: FOLLOWER
I20260925 02:57:01.326243 22912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d192ed873304f7698875e605d417939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d192ed873304f7698875e605d417939" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33103 } }
I20260925 02:57:01.326485 22912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d192ed873304f7698875e605d417939 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:57:01.326619 22912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d192ed873304f7698875e605d417939 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:57:01.326781 22912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d192ed873304f7698875e605d417939 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:01.329031 22912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d192ed873304f7698875e605d417939 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d192ed873304f7698875e605d417939" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33103 } }
I20260925 02:57:01.329499 22912 leader_election.cc:302] T 00000000000000000000000000000000 P 2d192ed873304f7698875e605d417939 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d192ed873304f7698875e605d417939; no voters:
I20260925 02:57:01.330029 22912 leader_election.cc:288] T 00000000000000000000000000000000 P 2d192ed873304f7698875e605d417939 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:57:01.330163 22917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d192ed873304f7698875e605d417939 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:01.330866 22917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d192ed873304f7698875e605d417939 [term 1 LEADER]: Becoming Leader. State: Replica: 2d192ed873304f7698875e605d417939, State: Running, Role: LEADER
I20260925 02:57:01.331409 22917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d192ed873304f7698875e605d417939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d192ed873304f7698875e605d417939" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33103 } }
I20260925 02:57:01.332036 22912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d192ed873304f7698875e605d417939 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:01.334374 22918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d192ed873304f7698875e605d417939 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d192ed873304f7698875e605d417939" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d192ed873304f7698875e605d417939" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33103 } } }
I20260925 02:57:01.334971 22918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d192ed873304f7698875e605d417939 [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:01.334447 22919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d192ed873304f7698875e605d417939 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d192ed873304f7698875e605d417939. Latest consensus state: current_term: 1 leader_uuid: "2d192ed873304f7698875e605d417939" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d192ed873304f7698875e605d417939" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33103 } } }
I20260925 02:57:01.335201 22919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d192ed873304f7698875e605d417939 [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:01.336706 22926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:01.341614 22926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:01.350086 22926 catalog_manager.cc:1347] Generated new cluster ID: 2c597e7bf4744ff0937f4f29963c854a
I20260925 02:57:01.350198 22926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:01.375941 22926 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:57:01.378086 22926 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:01.397156 22926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d192ed873304f7698875e605d417939: Generated new TSK 0
I20260925 02:57:01.398154 22926 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
I20260925 02:57:01.448438 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33103
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:42171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:01.768805 22938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:01.769115 22938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:01.769302 22938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:01.781741 22938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:01.781912 22938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:01.782150 22938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 02:57:01.798454 22938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:42171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:01.800335 22938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:01.802187 22938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:01.819239 22946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:01.819185 22943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:01.819235 22944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:01.820343 22938 server_base.cc:1034] running on GCE node
I20260925 02:57:01.820883 22938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:01.821794 22938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:01.822986 22938 hybrid_clock.cc:648] HybridClock initialized: now 1790305021822937 us; error 26 us; skew 500 ppm
I20260925 02:57:01.823283 22938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:01 dist-test-slave-8zvk krb5kdc[22832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305021, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:01.826246 22938 init.cc:377] Logged in from keytab as kudu/127.18.196.193@KRBTEST.COM (short username kudu)
I20260925 02:57:01.827695 22938 webserver.cc:460] Webserver started at http://127.18.196.193:32801/ using document root <none> and password file <none>
I20260925 02:57:01.828347 22938 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:01.828490 22938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:01.828799 22938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:01.831772 22938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "dd7d2b0652c647948646799251457792"
format_stamp: "Formatted at 2026-09-25 02:57:01 on dist-test-slave-8zvk"
server_key: "19662515fb03276068964e4e9eddb0d0"
server_key_iv: "7bdca6952d80511b11bd9b2309c74e9a"
server_key_version: "encryptionkey@0"
I20260925 02:57:01.832633 22938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "dd7d2b0652c647948646799251457792"
format_stamp: "Formatted at 2026-09-25 02:57:01 on dist-test-slave-8zvk"
server_key: "19662515fb03276068964e4e9eddb0d0"
server_key_iv: "7bdca6952d80511b11bd9b2309c74e9a"
server_key_version: "encryptionkey@0"
I20260925 02:57:01.838037 22938 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 02:57:01.841353 22953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:01.842582 22938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 02:57:01.842748 22938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "dd7d2b0652c647948646799251457792"
format_stamp: "Formatted at 2026-09-25 02:57:01 on dist-test-slave-8zvk"
server_key: "19662515fb03276068964e4e9eddb0d0"
server_key_iv: "7bdca6952d80511b11bd9b2309c74e9a"
server_key_version: "encryptionkey@0"
I20260925 02:57:01.842990 22938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:01.859300 22938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:01.863556 22938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:01.863852 22938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:01.864737 22938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:01.866015 22938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:01.866113 22938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:01.866254 22938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:01.866330 22938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:01.898393 22938 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:33741
I20260925 02:57:01.898442 23065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:33741 every 8 connection(s)
I20260925 02:57:01.899932 22938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 02:57:01.908551 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 22938
I20260925 02:57:01.908768 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 02:57:01.909391 19219 external_mini_cluster.cc:1428] Setting key 334c0f3fd1290d4a42bc6464b4f79afa
Sep 25 02:57:01 dist-test-slave-8zvk krb5kdc[22832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305021, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:57:01.921018 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:01.902368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:34753 (local address 127.18.196.254:33103)
0925 02:57:01.902773 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:01.902777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:01.903789 (+ 1012us) server_negotiation.cc:407] Connection header received
0925 02:57:01.905588 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:01.905594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:01.905679 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:01.905792 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:01.907256 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:01.907985 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:01.909595 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:01.910043 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:01.910187 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:01.914063 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:01.914072 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:01.914080 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:01.914126 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:01.917577 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:01.918625 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:01.918629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:01.918631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:01.918748 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:01.919302 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:01.919307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:01.919309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:01.919602 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:01.919720 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:01.920587 (+ 867us) server_negotiation.cc:299] Negotiation successful
0925 02:57:01.920757 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":206,"threads_started":1}
I20260925 02:57:01.922511 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:01.902978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:33103 (local address 127.18.196.193:34753)
0925 02:57:01.903641 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:01.903677 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:01.905328 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:01.906019 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:01.906033 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:01.906506 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:01.907096 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:01.907115 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:01.908148 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:01.908152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:01.909449 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:01.909461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:01.910323 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:01.910332 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:01.910594 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:01.911303 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:01.911337 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:01.913837 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:01.917817 (+ 3980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:01.917830 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:01.917850 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:01.918482 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:01.918885 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:01.918892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:01.918895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:01.919156 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:01.919728 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:01.919739 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:01.920182 (+ 443us) client_negotiation.cc:769] Sending connection context
0925 02:57:01.920475 (+ 293us) client_negotiation.cc:240] Negotiation successful
0925 02:57:01.921045 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":251,"threads_started":1}
I20260925 02:57:01.924176 23066 heartbeater.cc:347] Connected to a master server at 127.18.196.254:33103
I20260925 02:57:01.924495 23066 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:01.925338 23066 heartbeater.cc:511] Master 127.18.196.254:33103 requested a full tablet report, sending...
I20260925 02:57:01.927501 22877 ts_manager.cc:194] Registered new tserver with Master: dd7d2b0652c647948646799251457792 (127.18.196.193:33741)
I20260925 02:57:01.928906 22877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.193@KRBTEST.COM'} at 127.18.196.193:34753
WARNING: no policy specified for kudu/127.18.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
I20260925 02:57:01.959237 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33103
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:42171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:02.277052 23072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:02.277314 23072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:02.277407 23072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:02.284447 23072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:02.284551 23072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:02.284660 23072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 02:57:02.294106 23072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:42171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:02.295316 23072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:02.296655 23072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:02.309505 23077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:02.309665 23080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:02.310112 23078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:02.310590 23072 server_base.cc:1034] running on GCE node
I20260925 02:57:02.311261 23072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:02.312125 23072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:02.313339 23072 hybrid_clock.cc:648] HybridClock initialized: now 1790305022313295 us; error 37 us; skew 500 ppm
I20260925 02:57:02.313639 23072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:02 dist-test-slave-8zvk krb5kdc[22832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305022, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:02.316592 23072 init.cc:377] Logged in from keytab as kudu/127.18.196.194@KRBTEST.COM (short username kudu)
I20260925 02:57:02.317938 23072 webserver.cc:460] Webserver started at http://127.18.196.194:37791/ using document root <none> and password file <none>
I20260925 02:57:02.318593 23072 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:02.318710 23072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:02.319104 23072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:02.321982 23072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "75e00b44c50642b98b15c87a5fefba1f"
format_stamp: "Formatted at 2026-09-25 02:57:02 on dist-test-slave-8zvk"
server_key: "f0b11f10d2bec08e7e0d36680be07dfa"
server_key_iv: "1e61c04d5fc060fdf9a1b789f4752e3c"
server_key_version: "encryptionkey@0"
I20260925 02:57:02.322772 23072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "75e00b44c50642b98b15c87a5fefba1f"
format_stamp: "Formatted at 2026-09-25 02:57:02 on dist-test-slave-8zvk"
server_key: "f0b11f10d2bec08e7e0d36680be07dfa"
server_key_iv: "1e61c04d5fc060fdf9a1b789f4752e3c"
server_key_version: "encryptionkey@0"
I20260925 02:57:02.327710 23072 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 02:57:02.330991 23087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:02.332360 23072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:57:02.332477 23072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "75e00b44c50642b98b15c87a5fefba1f"
format_stamp: "Formatted at 2026-09-25 02:57:02 on dist-test-slave-8zvk"
server_key: "f0b11f10d2bec08e7e0d36680be07dfa"
server_key_iv: "1e61c04d5fc060fdf9a1b789f4752e3c"
server_key_version: "encryptionkey@0"
I20260925 02:57:02.332690 23072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:02.365363 23072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:02.369024 23072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:02.369298 23072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:02.370184 23072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:02.371485 23072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:02.371557 23072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:02.371719 23072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:02.371767 23072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:02.403438 23072 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:38385
I20260925 02:57:02.403540 23199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:38385 every 8 connection(s)
I20260925 02:57:02.404673 23072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 02:57:02.407791 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 23072
I20260925 02:57:02.407995 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 02:57:02.408560 19219 external_mini_cluster.cc:1428] Setting key da9b353af894eaa454271c4221ca57d0
Sep 25 02:57:02 dist-test-slave-8zvk krb5kdc[22832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305022, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:57:02.426005 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:02.407118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:57961 (local address 127.18.196.254:33103)
0925 02:57:02.407283 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:02.407287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:02.408527 (+ 1240us) server_negotiation.cc:407] Connection header received
0925 02:57:02.410789 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:02.410793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:02.410907 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:02.410988 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:02.413098 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:02.413839 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:02.416390 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:02.416826 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:02.416935 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:02.420354 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:02.420359 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:02.420362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:02.420399 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:02.423034 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:02.423723 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:02.423729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:02.423732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:02.423858 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:02.424273 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:02.424277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:02.424285 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:02.424540 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:02.424700 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:02.425448 (+ 748us) server_negotiation.cc:299] Negotiation successful
0925 02:57:02.425672 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:57:02.427021 23202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:02.407552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:33103 (local address 127.18.196.194:57961)
0925 02:57:02.408338 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:02.408401 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:02.410506 (+ 2105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:02.411267 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:02.411280 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:02.412060 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:02.412877 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:02.412939 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:02.414003 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:02.414011 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:02.416182 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:02.416200 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:02.417074 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:02.417080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:02.417288 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:02.417966 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:02.417991 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:02.420128 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:02.423165 (+ 3037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:02.423173 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:02.423189 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:02.423596 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:02.423968 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:02.423973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:02.423976 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:02.424157 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:02.424667 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:02.424675 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:02.425080 (+ 405us) client_negotiation.cc:769] Sending connection context
0925 02:57:02.425331 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 02:57:02.425829 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":304,"threads_started":1}
I20260925 02:57:02.428598 23200 heartbeater.cc:347] Connected to a master server at 127.18.196.254:33103
I20260925 02:57:02.428882 23200 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.18.196.195@KRBTEST.COM; defaulting to no policy
I20260925 02:57:02.429571 23200 heartbeater.cc:511] Master 127.18.196.254:33103 requested a full tablet report, sending...
I20260925 02:57:02.431097 22877 ts_manager.cc:194] Registered new tserver with Master: 75e00b44c50642b98b15c87a5fefba1f (127.18.196.194:38385)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.195@KRBTEST.COM" created.
I20260925 02:57:02.432092 22877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.194@KRBTEST.COM'} at 127.18.196.194:57961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
I20260925 02:57:02.455600 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33103
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:42171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:02.752676 23205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:02.752900 23205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:02.752992 23205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:02.759596 23205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:02.759680 23205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:02.759778 23205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 02:57:02.768548 23205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:42171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:02.769536 23205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:02.770617 23205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:02.782876 23213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:02.782821 23211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:02.782917 23210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:02.788070 23205 server_base.cc:1034] running on GCE node
I20260925 02:57:02.788573 23205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:02.789377 23205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:02.790591 23205 hybrid_clock.cc:648] HybridClock initialized: now 1790305022790550 us; error 30 us; skew 500 ppm
I20260925 02:57:02.790938 23205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:02 dist-test-slave-8zvk krb5kdc[22832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305022, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:02.793771 23205 init.cc:377] Logged in from keytab as kudu/127.18.196.195@KRBTEST.COM (short username kudu)
I20260925 02:57:02.795226 23205 webserver.cc:460] Webserver started at http://127.18.196.195:40467/ using document root <none> and password file <none>
I20260925 02:57:02.795864 23205 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:02.796001 23205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:02.796284 23205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:02.799015 23205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "6484a2df5c5d472e845fb597959be1dc"
format_stamp: "Formatted at 2026-09-25 02:57:02 on dist-test-slave-8zvk"
server_key: "96bf4a3f1f86ef9bbcbe1a694a09895b"
server_key_iv: "7d54824d03decfc6184fb5787bdb7ce2"
server_key_version: "encryptionkey@0"
I20260925 02:57:02.799762 23205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "6484a2df5c5d472e845fb597959be1dc"
format_stamp: "Formatted at 2026-09-25 02:57:02 on dist-test-slave-8zvk"
server_key: "96bf4a3f1f86ef9bbcbe1a694a09895b"
server_key_iv: "7d54824d03decfc6184fb5787bdb7ce2"
server_key_version: "encryptionkey@0"
I20260925 02:57:02.804548 23205 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:57:02.807699 23220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:02.808851 23205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:57:02.809022 23205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "6484a2df5c5d472e845fb597959be1dc"
format_stamp: "Formatted at 2026-09-25 02:57:02 on dist-test-slave-8zvk"
server_key: "96bf4a3f1f86ef9bbcbe1a694a09895b"
server_key_iv: "7d54824d03decfc6184fb5787bdb7ce2"
server_key_version: "encryptionkey@0"
I20260925 02:57:02.809224 23205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:02.840245 23205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:02.844472 23205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:02.844753 23205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:02.845672 23205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:02.847023 23205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:02.847121 23205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:02.847268 23205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:02.847342 23205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:02.882413 23205 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:40717
I20260925 02:57:02.882491 23332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:40717 every 8 connection(s)
I20260925 02:57:02.884272 23205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 02:57:02.892762 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 23205
I20260925 02:57:02.892992 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 02:57:02.893563 19219 external_mini_cluster.cc:1428] Setting key bc95601535acc5b1969430436023a371
Sep 25 02:57:02 dist-test-slave-8zvk krb5kdc[22832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305022, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:57:02.904757 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:02.887154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:32967 (local address 127.18.196.254:33103)
0925 02:57:02.887354 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:02.887360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:02.888556 (+ 1196us) server_negotiation.cc:407] Connection header received
0925 02:57:02.890316 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:02.890320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:02.890400 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:02.890494 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:02.891976 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:02.892778 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:02.894962 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:02.895429 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:02.895564 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:02.899191 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:02.899196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:02.899199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:02.899230 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:02.901878 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:02.902550 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:02.902555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:02.902559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:02.902696 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:02.903145 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:02.903148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:02.903150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:02.903401 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:02.903546 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:02.904208 (+ 662us) server_negotiation.cc:299] Negotiation successful
0925 02:57:02.904415 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 02:57:02.905594 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:02.887647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:33103 (local address 127.18.196.195:32967)
0925 02:57:02.888404 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:02.888444 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:02.890071 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:02.890668 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:02.890676 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:02.891252 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:02.891799 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:02.891815 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:02.892983 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:02.892990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:02.894794 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:02.894806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:02.895691 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:02.895699 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:02.896006 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:02.896756 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:02.896778 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:02.898976 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:02.902001 (+ 3025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:02.902010 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:02.902024 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:02.902419 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:02.902799 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:02.902804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:02.902807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:02.903025 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:02.903520 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:02.903527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:02.903848 (+ 321us) client_negotiation.cc:769] Sending connection context
0925 02:57:02.904103 (+ 255us) client_negotiation.cc:240] Negotiation successful
0925 02:57:02.904592 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":337,"threads_started":1}
I20260925 02:57:02.906808 23333 heartbeater.cc:347] Connected to a master server at 127.18.196.254:33103
I20260925 02:57:02.907147 23333 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:02.907971 23333 heartbeater.cc:511] Master 127.18.196.254:33103 requested a full tablet report, sending...
I20260925 02:57:02.909354 22877 ts_manager.cc:194] Registered new tserver with Master: 6484a2df5c5d472e845fb597959be1dc (127.18.196.195:40717)
I20260925 02:57:02.910146 22877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.195@KRBTEST.COM'} at 127.18.196.195:32967
I20260925 02:57:02.920543 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:57:02.932120 23066 heartbeater.cc:503] Master 127.18.196.254:33103 was elected leader, sending a full tablet report...
Sep 25 02:57:02 dist-test-slave-8zvk krb5kdc[22832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:57:02 dist-test-slave-8zvk krb5kdc[22832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.193@KRBTEST.COM
I20260925 02:57:02.968716 23340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:02.953886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59136 (local address 127.18.196.193:33741)
0925 02:57:02.954532 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:02.954543 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:02.954576 (+ 33us) server_negotiation.cc:407] Connection header received
0925 02:57:02.954653 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:02.954659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:02.954939 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:02.955106 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:02.956469 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:02.957528 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:02.958743 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:02.959332 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:02.959465 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:02.963233 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:02.963243 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:02.963255 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:02.963282 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:02.966015 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:02.966649 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:02.966656 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:02.966658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:02.966776 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:02.967158 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:02.967164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:02.967167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:02.967500 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:02.967801 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:02.968107 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 02:57:02.968361 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":300,"threads_started":1}
Sep 25 02:57:02 dist-test-slave-8zvk krb5kdc[22832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:57:02.985672 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:02.972542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55122 (local address 127.18.196.254:33103)
0925 02:57:02.972741 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:02.972747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:02.972906 (+ 159us) server_negotiation.cc:407] Connection header received
0925 02:57:02.973091 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:02.973095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:02.973154 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:02.973246 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:02.974479 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:02.975302 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:02.976530 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:02.976924 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:02.977057 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:02.980638 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:02.980643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:02.980646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:02.980676 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:02.983182 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:02.983759 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:02.983784 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:02.983788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:02.983912 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:02.984267 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:02.984272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:02.984275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:02.984590 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:02.984844 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:02.985159 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 02:57:02.985365 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 02:57:02.986178 22877 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55122
I20260925 02:57:02.987058 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 22938
I20260925 02:57:02.998396 23340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:02.987942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59140 (local address 127.18.196.193:33741)
0925 02:57:02.988136 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:02.988141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:02.988224 (+ 83us) server_negotiation.cc:407] Connection header received
0925 02:57:02.988352 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:02.988355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:02.988414 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:02.988504 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:02.989586 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:02.990220 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:02.991419 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:02.991816 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:02.991919 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:02.993938 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:02.993942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:02.993945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:02.993968 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:02.996349 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:02.996867 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:02.996871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:02.996872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:02.996989 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:02.997278 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:02.997280 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:02.997282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:02.997492 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:02.997682 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:02.997912 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 02:57:02.998069 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:57:02.999014 23061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:03.080503 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 22938
I20260925 02:57:03.097375 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 23072
Sep 25 02:57:03 dist-test-slave-8zvk krb5kdc[22832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.194@KRBTEST.COM
I20260925 02:57:03.113065 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:03.098303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57698 (local address 127.18.196.194:38385)
0925 02:57:03.098818 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:03.098825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:03.098902 (+ 77us) server_negotiation.cc:407] Connection header received
0925 02:57:03.098965 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:03.098972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:03.099182 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:03.099346 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:03.100564 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:03.101550 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:03.102872 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:03.103422 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:03.103559 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:03.107538 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:03.107549 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:03.107563 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:03.107596 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:03.110453 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:03.111115 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:03.111122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:03.111124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:03.111239 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:03.111612 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:03.111616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:03.111618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:03.111937 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:03.112221 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:03.112458 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 02:57:03.112704 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":255,"threads_started":1}
I20260925 02:57:03.114292 23195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:03.196162 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 23072
I20260925 02:57:03.212440 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 23205
Sep 25 02:57:03 dist-test-slave-8zvk krb5kdc[22832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.195@KRBTEST.COM
I20260925 02:57:03.231285 23347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:03.213718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36982 (local address 127.18.196.195:40717)
0925 02:57:03.214574 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:03.214585 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:03.214625 (+ 40us) server_negotiation.cc:407] Connection header received
0925 02:57:03.214713 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:03.214724 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:03.215092 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:03.215281 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:03.217191 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:03.218702 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:03.220591 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:03.221517 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:03.221620 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:03.225991 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:03.226002 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:03.226016 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:03.226048 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:03.228939 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:03.229565 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:03.229571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:03.229574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:03.229691 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:03.229967 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:03.229970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:03.229972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:03.230279 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:03.230522 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:03.230721 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 02:57:03.230998 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":469,"threads_started":1}
I20260925 02:57:03.232702 23328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:03.314962 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 23205
I20260925 02:57:03.331584 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 22845
I20260925 02:57:03.332820 22907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:03.435132 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 22845
2026-09-25T02:57:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3804 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:57:03 dist-test-slave-8zvk krb5kdc[23352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:57:03 dist-test-slave-8zvk krb5kdc[23352](info): set up 2 sockets
Sep 25 02:57:03 dist-test-slave-8zvk krb5kdc[23352](info): commencing operation
krb5kdc: starting...
W20260925 02:57:04.490260 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 02:57:04 dist-test-slave-8zvk krb5kdc[23352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:57:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:57:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 02:57:04.671756 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:44403
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:33449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:44403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:04.972728 23365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:04.972967 23365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:04.973073 23365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:04.979395 23365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:04.979477 23365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:04.979568 23365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:04.979661 23365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:04.979751 23365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:04.988907 23365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:33449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:44403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:44403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:04.990067 23365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:04.991391 23365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:05.001974 23373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:05.002027 23371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:05.002056 23370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:05.003058 23365 server_base.cc:1034] running on GCE node
I20260925 02:57:05.003765 23365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:05.005051 23365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:05.006290 23365 hybrid_clock.cc:648] HybridClock initialized: now 1790305025006264 us; error 49 us; skew 500 ppm
I20260925 02:57:05.006601 23365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:05 dist-test-slave-8zvk krb5kdc[23352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305025, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:05.009315 23365 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 02:57:05.010249 23365 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:05.010375 23365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:05.010660 23365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:05.013283 23365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "389fbf50a8b04d18ad2c7359aa73b9c0"
format_stamp: "Formatted at 2026-09-25 02:57:05 on dist-test-slave-8zvk"
server_key: "d364942b3205c8f82d46146a0aa69250"
server_key_iv: "7e91bb249868729e445d4dc5d9bd1a50"
server_key_version: "encryptionkey@0"
I20260925 02:57:05.014038 23365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "389fbf50a8b04d18ad2c7359aa73b9c0"
format_stamp: "Formatted at 2026-09-25 02:57:05 on dist-test-slave-8zvk"
server_key: "d364942b3205c8f82d46146a0aa69250"
server_key_iv: "7e91bb249868729e445d4dc5d9bd1a50"
server_key_version: "encryptionkey@0"
I20260925 02:57:05.018471 23365 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 02:57:05.021695 23379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:05.022993 23365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 02:57:05.023190 23365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "389fbf50a8b04d18ad2c7359aa73b9c0"
format_stamp: "Formatted at 2026-09-25 02:57:05 on dist-test-slave-8zvk"
server_key: "d364942b3205c8f82d46146a0aa69250"
server_key_iv: "7e91bb249868729e445d4dc5d9bd1a50"
server_key_version: "encryptionkey@0"
I20260925 02:57:05.023427 23365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:05.061337 23365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:05.065326 23365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:05.065604 23365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:05.083753 23365 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:44403
I20260925 02:57:05.083777 23430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:44403 every 8 connection(s)
I20260925 02:57:05.084633 23365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:57:05.085949 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 23365
I20260925 02:57:05.086163 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:57:05.086967 19219 external_mini_cluster.cc:1428] Setting key f94ebe01182fe2d2076c3e40208cb87a
I20260925 02:57:05.089203 23431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 02:57:05 dist-test-slave-8zvk krb5kdc[23352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:57:05.102484 23431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 389fbf50a8b04d18ad2c7359aa73b9c0: Bootstrap starting.
I20260925 02:57:05.105960 23431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 389fbf50a8b04d18ad2c7359aa73b9c0: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:05.107044 23431 log.cc:824] T 00000000000000000000000000000000 P 389fbf50a8b04d18ad2c7359aa73b9c0: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:05.110090 23431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 389fbf50a8b04d18ad2c7359aa73b9c0: No bootstrap required, opened a new log
I20260925 02:57:05.110328 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:05.089027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33676 (local address 127.18.196.254:44403)
0925 02:57:05.089829 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:05.089845 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:05.089900 (+ 55us) server_negotiation.cc:407] Connection header received
0925 02:57:05.092102 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:05.092127 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:05.092635 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:05.093056 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:05.094335 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:05.096406 (+ 2071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:05.097643 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:05.098694 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:05.098926 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:05.103139 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:05.103147 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:05.103160 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:05.103198 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:05.106089 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:05.106766 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:05.106780 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:05.106786 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:05.107002 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:05.107401 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:05.107407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:05.107411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:05.108199 (+ 788us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:05.108459 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:05.108958 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 02:57:05.109322 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":339,"threads_started":1}
I20260925 02:57:05.118043 23431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 389fbf50a8b04d18ad2c7359aa73b9c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389fbf50a8b04d18ad2c7359aa73b9c0" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 44403 } }
I20260925 02:57:05.118413 23431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 389fbf50a8b04d18ad2c7359aa73b9c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:05.118518 23431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 389fbf50a8b04d18ad2c7359aa73b9c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 389fbf50a8b04d18ad2c7359aa73b9c0, State: Initialized, Role: FOLLOWER
I20260925 02:57:05.119112 23431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 389fbf50a8b04d18ad2c7359aa73b9c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389fbf50a8b04d18ad2c7359aa73b9c0" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 44403 } }
I20260925 02:57:05.119369 23431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 389fbf50a8b04d18ad2c7359aa73b9c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:57:05.119515 23431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 389fbf50a8b04d18ad2c7359aa73b9c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:57:05.119686 23431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 389fbf50a8b04d18ad2c7359aa73b9c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:05.121785 23431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 389fbf50a8b04d18ad2c7359aa73b9c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389fbf50a8b04d18ad2c7359aa73b9c0" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 44403 } }
I20260925 02:57:05.122259 23431 leader_election.cc:302] T 00000000000000000000000000000000 P 389fbf50a8b04d18ad2c7359aa73b9c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 389fbf50a8b04d18ad2c7359aa73b9c0; no voters:
I20260925 02:57:05.122901 23431 leader_election.cc:288] T 00000000000000000000000000000000 P 389fbf50a8b04d18ad2c7359aa73b9c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:57:05.123057 23437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 389fbf50a8b04d18ad2c7359aa73b9c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:05.124212 23437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 389fbf50a8b04d18ad2c7359aa73b9c0 [term 1 LEADER]: Becoming Leader. State: Replica: 389fbf50a8b04d18ad2c7359aa73b9c0, State: Running, Role: LEADER
I20260925 02:57:05.124683 23437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 389fbf50a8b04d18ad2c7359aa73b9c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389fbf50a8b04d18ad2c7359aa73b9c0" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 44403 } }
I20260925 02:57:05.125090 23431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 389fbf50a8b04d18ad2c7359aa73b9c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:05.127283 23439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389fbf50a8b04d18ad2c7359aa73b9c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 389fbf50a8b04d18ad2c7359aa73b9c0. Latest consensus state: current_term: 1 leader_uuid: "389fbf50a8b04d18ad2c7359aa73b9c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389fbf50a8b04d18ad2c7359aa73b9c0" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 44403 } } }
I20260925 02:57:05.127686 23439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389fbf50a8b04d18ad2c7359aa73b9c0 [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:05.128535 23438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389fbf50a8b04d18ad2c7359aa73b9c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "389fbf50a8b04d18ad2c7359aa73b9c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389fbf50a8b04d18ad2c7359aa73b9c0" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 44403 } } }
I20260925 02:57:05.128912 23438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389fbf50a8b04d18ad2c7359aa73b9c0 [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:05.133201 23446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:05.137506 23446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:05.145304 23446 catalog_manager.cc:1347] Generated new cluster ID: c33bb9525fce4be091ce93e9d3ba26ba
I20260925 02:57:05.145466 23446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:05.182116 23446 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:57:05.183555 23446 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:05.197691 23446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 389fbf50a8b04d18ad2c7359aa73b9c0: Generated new TSK 0
I20260925 02:57:05.198490 23446 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
I20260925 02:57:05.258873 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:44403
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:33449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:05.591296 23458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:05.591540 23458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:05.591739 23458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:05.599179 23458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:05.599295 23458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:05.599505 23458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 02:57:05.608772 23458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:33449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:44403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:05.609897 23458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:05.611246 23458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:05.623324 23463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:05.623296 23464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:05.623373 23466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:05.624089 23458 server_base.cc:1034] running on GCE node
I20260925 02:57:05.624665 23458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:05.625520 23458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:05.626711 23458 hybrid_clock.cc:648] HybridClock initialized: now 1790305025626669 us; error 50 us; skew 500 ppm
I20260925 02:57:05.627022 23458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:05 dist-test-slave-8zvk krb5kdc[23352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305025, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:05.629813 23458 init.cc:377] Logged in from keytab as kudu/127.18.196.193@KRBTEST.COM (short username kudu)
I20260925 02:57:05.630836 23458 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:05.631021 23458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:05.631309 23458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:05.634161 23458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "c5d9291a7d864d4289b402d96e4dc501"
format_stamp: "Formatted at 2026-09-25 02:57:05 on dist-test-slave-8zvk"
server_key: "07949145f4ec2bbc77f3c65b200b79bd"
server_key_iv: "7f9226a3ef96686b928ddd7bb71e39f8"
server_key_version: "encryptionkey@0"
I20260925 02:57:05.634974 23458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "c5d9291a7d864d4289b402d96e4dc501"
format_stamp: "Formatted at 2026-09-25 02:57:05 on dist-test-slave-8zvk"
server_key: "07949145f4ec2bbc77f3c65b200b79bd"
server_key_iv: "7f9226a3ef96686b928ddd7bb71e39f8"
server_key_version: "encryptionkey@0"
I20260925 02:57:05.639393 23458 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:57:05.642491 23472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:05.643684 23458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260925 02:57:05.643833 23458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "c5d9291a7d864d4289b402d96e4dc501"
format_stamp: "Formatted at 2026-09-25 02:57:05 on dist-test-slave-8zvk"
server_key: "07949145f4ec2bbc77f3c65b200b79bd"
server_key_iv: "7f9226a3ef96686b928ddd7bb71e39f8"
server_key_version: "encryptionkey@0"
I20260925 02:57:05.644057 23458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:05.662801 23458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:05.666311 23458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:05.666533 23458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:05.667451 23458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:05.668591 23458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:05.668699 23458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:05.668812 23458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:05.668896 23458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:05.699306 23458 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:45787
I20260925 02:57:05.699379 23584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:45787 every 8 connection(s)
I20260925 02:57:05.700053 23458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 02:57:05.709281 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 23458
I20260925 02:57:05.709496 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 02:57:05.710301 19219 external_mini_cluster.cc:1428] Setting key 2dbebb6fdec601965dd9ec710a215397
Sep 25 02:57:05 dist-test-slave-8zvk krb5kdc[23352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305025, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:57:05.720495 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:05.702348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:34377 (local address 127.18.196.254:44403)
0925 02:57:05.702779 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:05.702785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:05.703727 (+ 942us) server_negotiation.cc:407] Connection header received
0925 02:57:05.705434 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:05.705438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:05.705518 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:05.705613 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:05.706990 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:05.707671 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:05.709153 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:05.709825 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:05.709966 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:05.713768 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:05.713773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:05.713776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:05.713807 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:05.716930 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:05.717908 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:05.717916 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:05.717922 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:05.718068 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:05.718575 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:05.718585 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:05.718587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:05.718991 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:05.719147 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:05.719783 (+ 636us) server_negotiation.cc:299] Negotiation successful
0925 02:57:05.720027 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":202,"threads_started":1}
I20260925 02:57:05.720978 23588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:05.702948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:44403 (local address 127.18.196.193:34377)
0925 02:57:05.703576 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:05.703621 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:05.705168 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:05.705786 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:05.705794 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:05.706259 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:05.706769 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:05.706785 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:05.707804 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:05.707808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:05.708999 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:05.709011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:05.710015 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:05.710019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:05.710192 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:05.710894 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:05.710922 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:05.713570 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:05.717089 (+ 3519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:05.717099 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:05.717115 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:05.717763 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:05.718207 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:05.718214 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:05.718219 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:05.718440 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:05.719119 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:05.719125 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:05.719413 (+ 288us) client_negotiation.cc:769] Sending connection context
0925 02:57:05.719622 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 02:57:05.719962 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":255,"threads_started":1}
I20260925 02:57:05.722500 23585 heartbeater.cc:347] Connected to a master server at 127.18.196.254:44403
I20260925 02:57:05.722787 23585 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:05.723403 23585 heartbeater.cc:511] Master 127.18.196.254:44403 requested a full tablet report, sending...
I20260925 02:57:05.725476 23396 ts_manager.cc:194] Registered new tserver with Master: c5d9291a7d864d4289b402d96e4dc501 (127.18.196.193:45787)
I20260925 02:57:05.727212 23396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.193@KRBTEST.COM'} at 127.18.196.193:34377
WARNING: no policy specified for kudu/127.18.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
I20260925 02:57:05.755309 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:44403
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:33449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:06.067231 23591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:06.067543 23591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:06.067704 23591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:06.074677 23591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:06.074823 23591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:06.075013 23591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 02:57:06.084424 23591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:33449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:44403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:06.085610 23591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:06.086820 23591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:06.099440 23597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:06.099480 23599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:06.099649 23596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:06.100256 23591 server_base.cc:1034] running on GCE node
I20260925 02:57:06.100798 23591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:06.101676 23591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:06.102921 23591 hybrid_clock.cc:648] HybridClock initialized: now 1790305026102872 us; error 35 us; skew 500 ppm
I20260925 02:57:06.103255 23591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:06 dist-test-slave-8zvk krb5kdc[23352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305026, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:06.106169 23591 init.cc:377] Logged in from keytab as kudu/127.18.196.194@KRBTEST.COM (short username kudu)
I20260925 02:57:06.107191 23591 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:06.107304 23591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:06.107573 23591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:06.110209 23591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "eaac4df84d6a42da91c1ccac24dc67e4"
format_stamp: "Formatted at 2026-09-25 02:57:06 on dist-test-slave-8zvk"
server_key: "2d69ace3aa6d8485e9f6b81b6284e9e7"
server_key_iv: "782e53da4edb57e132a0e75054314a6b"
server_key_version: "encryptionkey@0"
I20260925 02:57:06.111030 23591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "eaac4df84d6a42da91c1ccac24dc67e4"
format_stamp: "Formatted at 2026-09-25 02:57:06 on dist-test-slave-8zvk"
server_key: "2d69ace3aa6d8485e9f6b81b6284e9e7"
server_key_iv: "782e53da4edb57e132a0e75054314a6b"
server_key_version: "encryptionkey@0"
I20260925 02:57:06.115743 23591 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:57:06.118947 23605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:06.120134 23591 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:57:06.120291 23591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "eaac4df84d6a42da91c1ccac24dc67e4"
format_stamp: "Formatted at 2026-09-25 02:57:06 on dist-test-slave-8zvk"
server_key: "2d69ace3aa6d8485e9f6b81b6284e9e7"
server_key_iv: "782e53da4edb57e132a0e75054314a6b"
server_key_version: "encryptionkey@0"
I20260925 02:57:06.120491 23591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:06.156255 23591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:06.160404 23591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:06.160652 23591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:06.161544 23591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:06.162791 23591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:06.162940 23591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:06.163074 23591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:06.163156 23591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:06.194469 23591 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:40565
I20260925 02:57:06.194548 23717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:40565 every 8 connection(s)
I20260925 02:57:06.195362 23591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 02:57:06.201990 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 23591
I20260925 02:57:06.202183 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 02:57:06.202718 19219 external_mini_cluster.cc:1428] Setting key 074386c98047aeafc3dc923148aec3cd
Sep 25 02:57:06 dist-test-slave-8zvk krb5kdc[23352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305026, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:57:06.215611 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:06.197776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:59521 (local address 127.18.196.254:44403)
0925 02:57:06.198010 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:06.198016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:06.198966 (+ 950us) server_negotiation.cc:407] Connection header received
0925 02:57:06.200666 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:06.200672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:06.200758 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:06.200860 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:06.202615 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.203555 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.205506 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.205927 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.206025 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:06.209999 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:06.210004 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:06.210007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:06.210038 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:06.212539 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:06.213365 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:06.213369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:06.213372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:06.213493 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:06.213894 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:06.213899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:06.213903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:06.214169 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:06.214325 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:06.215056 (+ 731us) server_negotiation.cc:299] Negotiation successful
0925 02:57:06.215280 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 02:57:06.216671 23720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:06.198144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:44403 (local address 127.18.196.194:59521)
0925 02:57:06.198779 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:06.198816 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:06.200406 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:06.201076 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:06.201084 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:06.201595 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:06.202425 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:06.202454 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.203703 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.203710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:06.205340 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:06.205365 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.206164 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.206173 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:06.206498 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:06.207232 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:06.207264 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:06.209782 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:06.212676 (+ 2894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:06.212685 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:06.212701 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:06.213224 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:06.213599 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:06.213604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:06.213608 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:06.213785 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:06.214306 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:06.214314 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:06.214650 (+ 336us) client_negotiation.cc:769] Sending connection context
0925 02:57:06.214941 (+ 291us) client_negotiation.cc:240] Negotiation successful
0925 02:57:06.215459 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":258,"threads_started":1}
I20260925 02:57:06.218158 23718 heartbeater.cc:347] Connected to a master server at 127.18.196.254:44403
I20260925 02:57:06.218510 23718 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:06.219408 23718 heartbeater.cc:511] Master 127.18.196.254:44403 requested a full tablet report, sending...
I20260925 02:57:06.220970 23396 ts_manager.cc:194] Registered new tserver with Master: eaac4df84d6a42da91c1ccac24dc67e4 (127.18.196.194:40565)
I20260925 02:57:06.221771 23396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.194@KRBTEST.COM'} at 127.18.196.194:59521
WARNING: no policy specified for kudu/127.18.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
I20260925 02:57:06.246160 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:44403
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:33449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:06.561254 23723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:06.561610 23723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:06.561713 23723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:06.568702 23723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:06.568804 23723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:06.568928 23723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 02:57:06.578543 23723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:33449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:44403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:06.579749 23723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:06.580914 23723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:06.593608 23728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:06.593567 23731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:06.593657 23729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:06.594383 23723 server_base.cc:1034] running on GCE node
I20260925 02:57:06.595023 23723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:06.595928 23723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:06.597154 23723 hybrid_clock.cc:648] HybridClock initialized: now 1790305026597106 us; error 45 us; skew 500 ppm
I20260925 02:57:06.597460 23723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:06 dist-test-slave-8zvk krb5kdc[23352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305026, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:06.604272 23723 init.cc:377] Logged in from keytab as kudu/127.18.196.195@KRBTEST.COM (short username kudu)
I20260925 02:57:06.605459 23723 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:06.605592 23723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:06.605912 23723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:06.608907 23723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "fa14a215b3a940a494a2797f127d9315"
format_stamp: "Formatted at 2026-09-25 02:57:06 on dist-test-slave-8zvk"
server_key: "0bb406c4ebccde0b9809b4a94d0db84a"
server_key_iv: "223a0760adcfd04bdb0a34bd5c9c3869"
server_key_version: "encryptionkey@0"
I20260925 02:57:06.609738 23723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "fa14a215b3a940a494a2797f127d9315"
format_stamp: "Formatted at 2026-09-25 02:57:06 on dist-test-slave-8zvk"
server_key: "0bb406c4ebccde0b9809b4a94d0db84a"
server_key_iv: "223a0760adcfd04bdb0a34bd5c9c3869"
server_key_version: "encryptionkey@0"
I20260925 02:57:06.614773 23723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260925 02:57:06.618062 23737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:06.619449 23723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:57:06.619609 23723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "fa14a215b3a940a494a2797f127d9315"
format_stamp: "Formatted at 2026-09-25 02:57:06 on dist-test-slave-8zvk"
server_key: "0bb406c4ebccde0b9809b4a94d0db84a"
server_key_iv: "223a0760adcfd04bdb0a34bd5c9c3869"
server_key_version: "encryptionkey@0"
I20260925 02:57:06.619822 23723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:06.635738 23723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:06.639683 23723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:06.639956 23723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:06.640851 23723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:06.642464 23723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:06.642597 23723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:06.642714 23723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:06.642798 23723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:06.673076 23723 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:46313
I20260925 02:57:06.673089 23849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:46313 every 8 connection(s)
I20260925 02:57:06.673892 23723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 02:57:06.683467 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 23723
I20260925 02:57:06.683648 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 02:57:06.684074 19219 external_mini_cluster.cc:1428] Setting key 219e2ceec1e6f421b2239e8367279260
Sep 25 02:57:06 dist-test-slave-8zvk krb5kdc[23352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305026, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:57:06.692101 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:06.676214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:59615 (local address 127.18.196.254:44403)
0925 02:57:06.676400 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:06.676404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:06.677427 (+ 1023us) server_negotiation.cc:407] Connection header received
0925 02:57:06.679201 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:06.679207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:06.679315 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:06.679442 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:06.680916 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.681653 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.683235 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.683695 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.683833 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:06.687303 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:06.687308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:06.687312 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:06.687343 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:06.689636 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:06.690265 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:06.690269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:06.690271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:06.690385 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:06.690710 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:06.690713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:06.690715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:06.691034 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:06.691143 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:06.691709 (+ 566us) server_negotiation.cc:299] Negotiation successful
0925 02:57:06.691858 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:57:06.692919 23852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:06.676615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:44403 (local address 127.18.196.195:59615)
0925 02:57:06.677277 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:06.677319 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:06.678940 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:06.679661 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:06.679669 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:06.680153 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:06.680739 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:06.680760 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.681805 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.681809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:06.683079 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:06.683092 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.683910 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.683919 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:06.684212 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:06.684852 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:06.684895 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:06.687124 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:06.689760 (+ 2636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:06.689766 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:06.689776 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:06.690167 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:06.690469 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:06.690472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:06.690474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:06.690622 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:06.691134 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:06.691139 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:06.691410 (+ 271us) client_negotiation.cc:769] Sending connection context
0925 02:57:06.691608 (+ 198us) client_negotiation.cc:240] Negotiation successful
0925 02:57:06.691939 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":273,"threads_started":1}
I20260925 02:57:06.694236 23850 heartbeater.cc:347] Connected to a master server at 127.18.196.254:44403
I20260925 02:57:06.694504 23850 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:06.695153 23850 heartbeater.cc:511] Master 127.18.196.254:44403 requested a full tablet report, sending...
I20260925 02:57:06.696475 23396 ts_manager.cc:194] Registered new tserver with Master: fa14a215b3a940a494a2797f127d9315 (127.18.196.195:46313)
I20260925 02:57:06.697199 23396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.195@KRBTEST.COM'} at 127.18.196.195:59615
I20260925 02:57:06.698995 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:57:06.715778 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:06.704587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.18.196.254:44403)
0925 02:57:06.704779 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:06.704787 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:06.705327 (+ 540us) server_negotiation.cc:407] Connection header received
0925 02:57:06.705506 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:06.705509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:06.705564 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:06.705634 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:06.707062 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.707723 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.708881 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.709249 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.709351 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:06.711435 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:06.711440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:06.711443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:06.711471 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:06.713602 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:06.714166 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:06.714169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:06.714171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:06.714283 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:06.714567 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:06.714570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:06.714572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:06.714885 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:06.715065 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:06.715323 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 02:57:06.715460 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:57:06.719632 23395 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:57:06.727684 23395 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:57:06.729974 23585 heartbeater.cc:503] Master 127.18.196.254:44403 was elected leader, sending a full tablet report...
I20260925 02:57:06.752353 23864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:06.739981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49482 (local address 127.18.196.195:46313)
0925 02:57:06.740712 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:06.740722 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:06.741284 (+ 562us) server_negotiation.cc:407] Connection header received
0925 02:57:06.741645 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:06.741651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:06.741914 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:06.742111 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:06.742767 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.744486 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.748247 (+ 3761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.750280 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.750504 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:06.750731 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:06.751611 (+ 880us) server_negotiation.cc:299] Negotiation successful
0925 02:57:06.751846 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":475,"threads_started":1}
I20260925 02:57:06.753618 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:06.739655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:46313 (local address 127.0.0.1:49482)
0925 02:57:06.741140 (+ 1485us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:06.741188 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:06.741443 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:06.742124 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:06.742131 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:06.742162 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:06.742561 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:06.742571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.744706 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.744714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:06.748018 (+ 3304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:06.748040 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.751004 (+ 2964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.751013 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:06.751282 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:06.751352 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 02:57:06.752852 (+ 1500us) client_negotiation.cc:240] Negotiation successful
0925 02:57:06.753130 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1253,"thread_start_us":1091,"threads_started":1}
I20260925 02:57:06.754818 23863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:06.739861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:45787 (local address 127.0.0.1:34422)
0925 02:57:06.743264 (+ 3403us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:06.743283 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:06.743631 (+ 348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:06.744725 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:06.744732 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:06.744766 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:06.745411 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:06.745427 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.748016 (+ 2589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.748024 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:06.750911 (+ 2887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:06.750932 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.753747 (+ 2815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.753756 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:06.754010 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:06.754057 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 02:57:06.754182 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 02:57:06.754364 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3275,"thread_start_us":3153,"threads_started":1}
I20260925 02:57:06.755584 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:06.740586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34422 (local address 127.18.196.193:45787)
0925 02:57:06.741219 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:06.741229 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:06.743529 (+ 2300us) server_negotiation.cc:407] Connection header received
0925 02:57:06.743875 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:06.743886 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:06.744219 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:06.744441 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:06.745659 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.747757 (+ 2098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.751167 (+ 3410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.753129 (+ 1962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.753364 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:06.753555 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:06.755124 (+ 1569us) server_negotiation.cc:299] Negotiation successful
0925 02:57:06.755291 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":326,"threads_started":1}
I20260925 02:57:06.757488 23785 tablet_service.cc:1467] Processing CreateTablet for tablet 76f645e5f59146aabfbf0cf0b39b7751 (DEFAULT_TABLE table=test-table [id=11bd313614334efc84c91b941f585e8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:57:06.758952 23785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f645e5f59146aabfbf0cf0b39b7751. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:06.762907 23867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:06.743651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41016 (local address 127.18.196.194:40565)
0925 02:57:06.756777 (+ 13126us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:06.756783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:06.756804 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:57:06.756876 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:06.756881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:06.757045 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:06.757224 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:06.757557 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.759065 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.760952 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.761854 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.762035 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:06.762211 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:06.762381 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 02:57:06.762532 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12993,"thread_start_us":13745,"threads_started":1}
I20260925 02:57:06.762639 23520 tablet_service.cc:1467] Processing CreateTablet for tablet 76f645e5f59146aabfbf0cf0b39b7751 (DEFAULT_TABLE table=test-table [id=11bd313614334efc84c91b941f585e8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:57:06.763305 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:06.743385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:40565 (local address 127.0.0.1:41016)
0925 02:57:06.754407 (+ 11022us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:06.754425 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:06.754519 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:06.757212 (+ 2693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:06.757217 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:06.757242 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:06.757389 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:06.757398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.759187 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.759192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:06.760792 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:06.760805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.762026 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.762031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:06.762195 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:06.762225 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 02:57:06.762337 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 02:57:06.763045 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10905,"thread_start_us":15522,"threads_started":1}
I20260925 02:57:06.764412 23520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f645e5f59146aabfbf0cf0b39b7751. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:06.766134 23653 tablet_service.cc:1467] Processing CreateTablet for tablet 76f645e5f59146aabfbf0cf0b39b7751 (DEFAULT_TABLE table=test-table [id=11bd313614334efc84c91b941f585e8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:57:06.767549 23653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f645e5f59146aabfbf0cf0b39b7751. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:06.774241 23868 tablet_bootstrap.cc:492] T 76f645e5f59146aabfbf0cf0b39b7751 P fa14a215b3a940a494a2797f127d9315: Bootstrap starting.
I20260925 02:57:06.777937 23868 tablet_bootstrap.cc:654] T 76f645e5f59146aabfbf0cf0b39b7751 P fa14a215b3a940a494a2797f127d9315: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:06.779214 23868 log.cc:824] T 76f645e5f59146aabfbf0cf0b39b7751 P fa14a215b3a940a494a2797f127d9315: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:06.779404 23869 tablet_bootstrap.cc:492] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501: Bootstrap starting.
I20260925 02:57:06.782606 23869 tablet_bootstrap.cc:654] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:06.783836 23870 tablet_bootstrap.cc:492] T 76f645e5f59146aabfbf0cf0b39b7751 P eaac4df84d6a42da91c1ccac24dc67e4: Bootstrap starting.
I20260925 02:57:06.783869 23869 log.cc:824] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:06.785872 23868 tablet_bootstrap.cc:492] T 76f645e5f59146aabfbf0cf0b39b7751 P fa14a215b3a940a494a2797f127d9315: No bootstrap required, opened a new log
I20260925 02:57:06.786125 23868 ts_tablet_manager.cc:1397] T 76f645e5f59146aabfbf0cf0b39b7751 P fa14a215b3a940a494a2797f127d9315: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260925 02:57:06.786471 23869 tablet_bootstrap.cc:492] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501: No bootstrap required, opened a new log
I20260925 02:57:06.786710 23869 ts_tablet_manager.cc:1397] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260925 02:57:06.789232 23870 tablet_bootstrap.cc:654] T 76f645e5f59146aabfbf0cf0b39b7751 P eaac4df84d6a42da91c1ccac24dc67e4: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:06.790434 23870 log.cc:824] T 76f645e5f59146aabfbf0cf0b39b7751 P eaac4df84d6a42da91c1ccac24dc67e4: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:06.793265 23870 tablet_bootstrap.cc:492] T 76f645e5f59146aabfbf0cf0b39b7751 P eaac4df84d6a42da91c1ccac24dc67e4: No bootstrap required, opened a new log
I20260925 02:57:06.793503 23870 ts_tablet_manager.cc:1397] T 76f645e5f59146aabfbf0cf0b39b7751 P eaac4df84d6a42da91c1ccac24dc67e4: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20260925 02:57:06.794276 23868 raft_consensus.cc:348] T 76f645e5f59146aabfbf0cf0b39b7751 P fa14a215b3a940a494a2797f127d9315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa14a215b3a940a494a2797f127d9315" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 46313 } } peers { permanent_uuid: "c5d9291a7d864d4289b402d96e4dc501" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 45787 } } peers { permanent_uuid: "eaac4df84d6a42da91c1ccac24dc67e4" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40565 } }
I20260925 02:57:06.794620 23868 raft_consensus.cc:374] T 76f645e5f59146aabfbf0cf0b39b7751 P fa14a215b3a940a494a2797f127d9315 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:06.794744 23868 raft_consensus.cc:729] T 76f645e5f59146aabfbf0cf0b39b7751 P fa14a215b3a940a494a2797f127d9315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa14a215b3a940a494a2797f127d9315, State: Initialized, Role: FOLLOWER
I20260925 02:57:06.795348 23868 consensus_queue.cc:260] T 76f645e5f59146aabfbf0cf0b39b7751 P fa14a215b3a940a494a2797f127d9315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa14a215b3a940a494a2797f127d9315" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 46313 } } peers { permanent_uuid: "c5d9291a7d864d4289b402d96e4dc501" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 45787 } } peers { permanent_uuid: "eaac4df84d6a42da91c1ccac24dc67e4" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40565 } }
I20260925 02:57:06.796396 23850 heartbeater.cc:503] Master 127.18.196.254:44403 was elected leader, sending a full tablet report...
I20260925 02:57:06.796746 23869 raft_consensus.cc:348] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa14a215b3a940a494a2797f127d9315" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 46313 } } peers { permanent_uuid: "c5d9291a7d864d4289b402d96e4dc501" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 45787 } } peers { permanent_uuid: "eaac4df84d6a42da91c1ccac24dc67e4" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40565 } }
I20260925 02:57:06.797140 23869 raft_consensus.cc:374] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:06.797225 23869 raft_consensus.cc:729] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d9291a7d864d4289b402d96e4dc501, State: Initialized, Role: FOLLOWER
I20260925 02:57:06.797900 23869 consensus_queue.cc:260] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa14a215b3a940a494a2797f127d9315" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 46313 } } peers { permanent_uuid: "c5d9291a7d864d4289b402d96e4dc501" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 45787 } } peers { permanent_uuid: "eaac4df84d6a42da91c1ccac24dc67e4" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40565 } }
I20260925 02:57:06.799597 23869 ts_tablet_manager.cc:1428] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260925 02:57:06.801571 23868 ts_tablet_manager.cc:1428] T 76f645e5f59146aabfbf0cf0b39b7751 P fa14a215b3a940a494a2797f127d9315: Time spent starting tablet: real 0.015s user 0.007s sys 0.004s
I20260925 02:57:06.803251 23870 raft_consensus.cc:348] T 76f645e5f59146aabfbf0cf0b39b7751 P eaac4df84d6a42da91c1ccac24dc67e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa14a215b3a940a494a2797f127d9315" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 46313 } } peers { permanent_uuid: "c5d9291a7d864d4289b402d96e4dc501" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 45787 } } peers { permanent_uuid: "eaac4df84d6a42da91c1ccac24dc67e4" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40565 } }
I20260925 02:57:06.803673 23870 raft_consensus.cc:374] T 76f645e5f59146aabfbf0cf0b39b7751 P eaac4df84d6a42da91c1ccac24dc67e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:06.803784 23870 raft_consensus.cc:729] T 76f645e5f59146aabfbf0cf0b39b7751 P eaac4df84d6a42da91c1ccac24dc67e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaac4df84d6a42da91c1ccac24dc67e4, State: Initialized, Role: FOLLOWER
I20260925 02:57:06.804338 23870 consensus_queue.cc:260] T 76f645e5f59146aabfbf0cf0b39b7751 P eaac4df84d6a42da91c1ccac24dc67e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa14a215b3a940a494a2797f127d9315" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 46313 } } peers { permanent_uuid: "c5d9291a7d864d4289b402d96e4dc501" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 45787 } } peers { permanent_uuid: "eaac4df84d6a42da91c1ccac24dc67e4" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40565 } }
I20260925 02:57:06.805215 23718 heartbeater.cc:503] Master 127.18.196.254:44403 was elected leader, sending a full tablet report...
I20260925 02:57:06.805606 23870 ts_tablet_manager.cc:1428] T 76f645e5f59146aabfbf0cf0b39b7751 P eaac4df84d6a42da91c1ccac24dc67e4: Time spent starting tablet: real 0.012s user 0.008s sys 0.006s
I20260925 02:57:06.835096 23875 raft_consensus.cc:482] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:57:06.835436 23875 raft_consensus.cc:504] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa14a215b3a940a494a2797f127d9315" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 46313 } } peers { permanent_uuid: "c5d9291a7d864d4289b402d96e4dc501" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 45787 } } peers { permanent_uuid: "eaac4df84d6a42da91c1ccac24dc67e4" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40565 } }
I20260925 02:57:06.837296 23875 leader_election.cc:288] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa14a215b3a940a494a2797f127d9315 (127.18.196.195:46313), eaac4df84d6a42da91c1ccac24dc67e4 (127.18.196.194:40565)
I20260925 02:57:06.844659 23867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:06.837932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:38521 (local address 127.18.196.194:40565)
0925 02:57:06.838095 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:06.838100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:06.838614 (+ 514us) server_negotiation.cc:407] Connection header received
0925 02:57:06.838975 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:06.838982 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:06.839072 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:06.839174 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:06.839818 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.840739 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.842678 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.843617 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.843784 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:06.843844 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:06.844262 (+ 418us) server_negotiation.cc:299] Negotiation successful
0925 02:57:06.844399 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:57:06.844987 23864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:06.837617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:48397 (local address 127.18.196.195:46313)
0925 02:57:06.837845 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:06.837854 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:06.839322 (+ 1468us) server_negotiation.cc:407] Connection header received
0925 02:57:06.839582 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:06.839590 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:06.839676 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:06.839776 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:06.840316 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.841137 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.842988 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.844030 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.844204 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:06.844280 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:06.844583 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 02:57:06.844725 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 02:57:06.845031 23878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:06.837704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:46313 (local address 127.18.196.193:48397)
0925 02:57:06.839206 (+ 1502us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:06.839226 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:06.839428 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:06.839805 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:06.839808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:06.839831 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:06.840162 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:06.840173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.841281 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.841285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:06.842814 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:06.842828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.844314 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.844319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:06.844432 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:06.844454 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:57:06.844529 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 02:57:06.844617 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1370,"mutex_wait_us":35,"spinlock_wait_cycles":18304,"thread_start_us":1209,"threads_started":1}
I20260925 02:57:06.845541 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:06.837596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:40565 (local address 127.18.196.193:38521)
0925 02:57:06.838464 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:06.838489 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:06.838670 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:06.839211 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:06.839215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:06.839238 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:06.839672 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:06.839686 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.840868 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.840873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:06.842530 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:06.842542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.843845 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.843851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:06.844036 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:06.844104 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 02:57:06.844955 (+ 851us) client_negotiation.cc:240] Negotiation successful
0925 02:57:06.845108 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":34,"spinlock_wait_cycles":26496,"thread_start_us":497,"threads_started":1}
I20260925 02:57:06.846259 23805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f645e5f59146aabfbf0cf0b39b7751" candidate_uuid: "c5d9291a7d864d4289b402d96e4dc501" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa14a215b3a940a494a2797f127d9315" is_pre_election: true
I20260925 02:57:06.846468 23673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f645e5f59146aabfbf0cf0b39b7751" candidate_uuid: "c5d9291a7d864d4289b402d96e4dc501" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaac4df84d6a42da91c1ccac24dc67e4" is_pre_election: true
I20260925 02:57:06.846904 23805 raft_consensus.cc:2454] T 76f645e5f59146aabfbf0cf0b39b7751 P fa14a215b3a940a494a2797f127d9315 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5d9291a7d864d4289b402d96e4dc501 in term 0.
I20260925 02:57:06.846982 23673 raft_consensus.cc:2454] T 76f645e5f59146aabfbf0cf0b39b7751 P eaac4df84d6a42da91c1ccac24dc67e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5d9291a7d864d4289b402d96e4dc501 in term 0.
I20260925 02:57:06.847664 23474 leader_election.cc:302] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c5d9291a7d864d4289b402d96e4dc501, eaac4df84d6a42da91c1ccac24dc67e4, fa14a215b3a940a494a2797f127d9315; no voters:
I20260925 02:57:06.848059 23875 raft_consensus.cc:2781] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:57:06.848202 23875 raft_consensus.cc:482] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:57:06.848310 23875 raft_consensus.cc:3037] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:06.850605 23875 raft_consensus.cc:504] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa14a215b3a940a494a2797f127d9315" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 46313 } } peers { permanent_uuid: "c5d9291a7d864d4289b402d96e4dc501" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 45787 } } peers { permanent_uuid: "eaac4df84d6a42da91c1ccac24dc67e4" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40565 } }
I20260925 02:57:06.851253 23875 leader_election.cc:288] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501 [CANDIDATE]: Term 1 election: Requested vote from peers fa14a215b3a940a494a2797f127d9315 (127.18.196.195:46313), eaac4df84d6a42da91c1ccac24dc67e4 (127.18.196.194:40565)
I20260925 02:57:06.851755 23805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f645e5f59146aabfbf0cf0b39b7751" candidate_uuid: "c5d9291a7d864d4289b402d96e4dc501" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa14a215b3a940a494a2797f127d9315"
I20260925 02:57:06.851867 23673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f645e5f59146aabfbf0cf0b39b7751" candidate_uuid: "c5d9291a7d864d4289b402d96e4dc501" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaac4df84d6a42da91c1ccac24dc67e4"
I20260925 02:57:06.851969 23805 raft_consensus.cc:3037] T 76f645e5f59146aabfbf0cf0b39b7751 P fa14a215b3a940a494a2797f127d9315 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:06.852105 23673 raft_consensus.cc:3037] T 76f645e5f59146aabfbf0cf0b39b7751 P eaac4df84d6a42da91c1ccac24dc67e4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:06.854159 23805 raft_consensus.cc:2454] T 76f645e5f59146aabfbf0cf0b39b7751 P fa14a215b3a940a494a2797f127d9315 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5d9291a7d864d4289b402d96e4dc501 in term 1.
I20260925 02:57:06.854341 23673 raft_consensus.cc:2454] T 76f645e5f59146aabfbf0cf0b39b7751 P eaac4df84d6a42da91c1ccac24dc67e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5d9291a7d864d4289b402d96e4dc501 in term 1.
I20260925 02:57:06.854579 23475 leader_election.cc:302] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5d9291a7d864d4289b402d96e4dc501, fa14a215b3a940a494a2797f127d9315; no voters:
I20260925 02:57:06.854913 23875 raft_consensus.cc:2781] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:06.855314 23875 raft_consensus.cc:686] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501 [term 1 LEADER]: Becoming Leader. State: Replica: c5d9291a7d864d4289b402d96e4dc501, State: Running, Role: LEADER
I20260925 02:57:06.855835 23875 consensus_queue.cc:237] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa14a215b3a940a494a2797f127d9315" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 46313 } } peers { permanent_uuid: "c5d9291a7d864d4289b402d96e4dc501" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 45787 } } peers { permanent_uuid: "eaac4df84d6a42da91c1ccac24dc67e4" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40565 } }
I20260925 02:57:06.860400 23395 catalog_manager.cc:5495] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5d9291a7d864d4289b402d96e4dc501 (127.18.196.193). New cstate: current_term: 1 leader_uuid: "c5d9291a7d864d4289b402d96e4dc501" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa14a215b3a940a494a2797f127d9315" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 46313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5d9291a7d864d4289b402d96e4dc501" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 45787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaac4df84d6a42da91c1ccac24dc67e4" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40565 } health_report { overall_health: UNKNOWN } } }
I20260925 02:57:06.892168 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:06.887282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34424 (local address 127.18.196.193:45787)
0925 02:57:06.887520 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:06.887526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:06.887637 (+ 111us) server_negotiation.cc:407] Connection header received
0925 02:57:06.887806 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:06.887810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:06.887884 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:06.888003 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:57:06.888308 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.889009 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.890193 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.890584 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.890670 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:06.891179 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:57:06.891663 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:57:06.891786 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:06.891843 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 02:57:06.891902 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:57:06.900071 23673 raft_consensus.cc:1261] T 76f645e5f59146aabfbf0cf0b39b7751 P eaac4df84d6a42da91c1ccac24dc67e4 [term 1 FOLLOWER]: Refusing update from remote peer c5d9291a7d864d4289b402d96e4dc501: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:57:06.900074 23805 raft_consensus.cc:1261] T 76f645e5f59146aabfbf0cf0b39b7751 P fa14a215b3a940a494a2797f127d9315 [term 1 FOLLOWER]: Refusing update from remote peer c5d9291a7d864d4289b402d96e4dc501: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:57:06.901114 23879 consensus_queue.cc:1035] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa14a215b3a940a494a2797f127d9315" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 46313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:57:06.901479 23875 consensus_queue.cc:1035] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaac4df84d6a42da91c1ccac24dc67e4" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:57:06.912338 23882 mvcc.cc:205] Tried to move back new op lower bound from 7333089390171758592 to 7333089390005481472. Current Snapshot: MvccSnapshot[applied={T|T < 7333089390171758592}]
I20260925 02:57:06.927981 23396 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:57:06.928267 23396 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:57:06.932010 23396 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 389fbf50a8b04d18ad2c7359aa73b9c0: Sending DeleteTablet for 3 replicas of tablet 76f645e5f59146aabfbf0cf0b39b7751
I20260925 02:57:06.933146 23785 tablet_service.cc:1514] Processing DeleteTablet for tablet 76f645e5f59146aabfbf0cf0b39b7751 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:06 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:49482
I20260925 02:57:06.933251 23653 tablet_service.cc:1514] Processing DeleteTablet for tablet 76f645e5f59146aabfbf0cf0b39b7751 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:06 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:41016
I20260925 02:57:06.933771 23520 tablet_service.cc:1514] Processing DeleteTablet for tablet 76f645e5f59146aabfbf0cf0b39b7751 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:06 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:34422
I20260925 02:57:06.936035 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 23458
I20260925 02:57:06.944386 23892 tablet_replica.cc:330] T 76f645e5f59146aabfbf0cf0b39b7751 P eaac4df84d6a42da91c1ccac24dc67e4: stopping tablet replica
I20260925 02:57:06.944924 23892 raft_consensus.cc:2229] T 76f645e5f59146aabfbf0cf0b39b7751 P eaac4df84d6a42da91c1ccac24dc67e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:57:06.945478 23892 raft_consensus.cc:2258] T 76f645e5f59146aabfbf0cf0b39b7751 P eaac4df84d6a42da91c1ccac24dc67e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:57:06.946414 23891 tablet_replica.cc:330] T 76f645e5f59146aabfbf0cf0b39b7751 P fa14a215b3a940a494a2797f127d9315: stopping tablet replica
I20260925 02:57:06.946880 23891 raft_consensus.cc:2229] T 76f645e5f59146aabfbf0cf0b39b7751 P fa14a215b3a940a494a2797f127d9315 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:57:06.947377 23891 raft_consensus.cc:2258] T 76f645e5f59146aabfbf0cf0b39b7751 P fa14a215b3a940a494a2797f127d9315 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:57:06.949689 23892 ts_tablet_manager.cc:1905] T 76f645e5f59146aabfbf0cf0b39b7751 P eaac4df84d6a42da91c1ccac24dc67e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:57:06.949694 23891 ts_tablet_manager.cc:1905] T 76f645e5f59146aabfbf0cf0b39b7751 P fa14a215b3a940a494a2797f127d9315: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:57:06.956732 23892 ts_tablet_manager.cc:1918] T 76f645e5f59146aabfbf0cf0b39b7751 P eaac4df84d6a42da91c1ccac24dc67e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:57:06.956959 23892 log.cc:1196] T 76f645e5f59146aabfbf0cf0b39b7751 P eaac4df84d6a42da91c1ccac24dc67e4: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/76f645e5f59146aabfbf0cf0b39b7751
I20260925 02:57:06.957556 23892 ts_tablet_manager.cc:1939] T 76f645e5f59146aabfbf0cf0b39b7751 P eaac4df84d6a42da91c1ccac24dc67e4: Deleting consensus metadata
I20260925 02:57:06.959517 23891 ts_tablet_manager.cc:1918] T 76f645e5f59146aabfbf0cf0b39b7751 P fa14a215b3a940a494a2797f127d9315: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:57:06.959522 23381 catalog_manager.cc:4841] TS eaac4df84d6a42da91c1ccac24dc67e4 (127.18.196.194:40565): tablet 76f645e5f59146aabfbf0cf0b39b7751 (table test-table [id=11bd313614334efc84c91b941f585e8f]) successfully deleted
I20260925 02:57:06.959678 23891 log.cc:1196] T 76f645e5f59146aabfbf0cf0b39b7751 P fa14a215b3a940a494a2797f127d9315: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/76f645e5f59146aabfbf0cf0b39b7751
I20260925 02:57:06.960268 23891 ts_tablet_manager.cc:1939] T 76f645e5f59146aabfbf0cf0b39b7751 P fa14a215b3a940a494a2797f127d9315: Deleting consensus metadata
I20260925 02:57:06.962040 23382 catalog_manager.cc:4841] TS fa14a215b3a940a494a2797f127d9315 (127.18.196.195:46313): tablet 76f645e5f59146aabfbf0cf0b39b7751 (table test-table [id=11bd313614334efc84c91b941f585e8f]) successfully deleted
I20260925 02:57:06.963030 23893 tablet_replica.cc:330] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501: stopping tablet replica
I20260925 02:57:06.963455 23893 raft_consensus.cc:2229] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:57:06.964167 23893 raft_consensus.cc:2258] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:57:06.966069 23893 ts_tablet_manager.cc:1905] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 02:57:06 dist-test-slave-8zvk krb5kdc[23352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.193@KRBTEST.COM
I20260925 02:57:06.972849 23893 ts_tablet_manager.cc:1918] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:57:06.973023 23893 log.cc:1196] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304987644417-19219-0/minicluster-data/ts-0/wal/wals/76f645e5f59146aabfbf0cf0b39b7751
I20260925 02:57:06.973564 23893 ts_tablet_manager.cc:1939] T 76f645e5f59146aabfbf0cf0b39b7751 P c5d9291a7d864d4289b402d96e4dc501: Deleting consensus metadata
I20260925 02:57:06.975438 23381 catalog_manager.cc:4841] TS c5d9291a7d864d4289b402d96e4dc501 (127.18.196.193:45787): tablet 76f645e5f59146aabfbf0cf0b39b7751 (table test-table [id=11bd313614334efc84c91b941f585e8f]) successfully deleted
I20260925 02:57:06.979750 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:06.937674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34434 (local address 127.18.196.193:45787)
0925 02:57:06.937898 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:06.937907 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:06.962194 (+ 24287us) server_negotiation.cc:407] Connection header received
0925 02:57:06.962639 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:06.962646 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:06.962754 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:06.962912 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:06.964498 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.965696 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.967496 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.968295 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.968436 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:06.973570 (+ 5134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:06.973584 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:06.973601 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:06.973650 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:06.977084 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:06.977876 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:06.977882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:06.977884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:06.978028 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:06.978405 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:06.978408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:06.978410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:06.978726 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:06.979001 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:06.979274 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 02:57:06.979461 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:57:06.980437 23580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:07.079505 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 23458
I20260925 02:57:07.103600 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 23591
Sep 25 02:57:07 dist-test-slave-8zvk krb5kdc[23352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.194@KRBTEST.COM
I20260925 02:57:07.119850 23867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:07.104434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41022 (local address 127.18.196.194:40565)
0925 02:57:07.104603 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:07.104607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:07.104625 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:57:07.104729 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:07.104733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:07.104792 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:07.104876 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:07.106369 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:07.107182 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:07.108905 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:07.109275 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:07.109362 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:07.114396 (+ 5034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:07.114410 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:07.114424 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:07.114467 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:07.117259 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:07.117870 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:07.117876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:07.117879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:07.118006 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:07.118324 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:07.118327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:07.118329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:07.118682 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:07.118944 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:07.119321 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 02:57:07.119498 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:57:07.120560 23713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:07.207861 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 23591
I20260925 02:57:07.232261 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 23723
Sep 25 02:57:07 dist-test-slave-8zvk krb5kdc[23352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.195@KRBTEST.COM
I20260925 02:57:07.248782 23864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:07.233089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49484 (local address 127.18.196.195:46313)
0925 02:57:07.233274 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:07.233280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:07.233323 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:57:07.233564 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:07.233570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:07.233658 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:07.233766 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:07.235099 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:07.236562 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:07.238027 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:07.238703 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:07.238873 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:07.243672 (+ 4799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:07.243687 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:07.243703 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:07.243738 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:07.246367 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:07.247009 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:07.247015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:07.247018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:07.247144 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:07.247493 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:07.247496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:07.247498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:07.247822 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:07.248025 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:07.248274 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 02:57:07.248431 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 02:57:07.249364 23845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:07.335989 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 23723
I20260925 02:57:07.353376 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 23365
I20260925 02:57:07.354470 23426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:07.461102 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 23365
2026-09-25T02:57:07Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4025 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T02:57:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:57:07Z Disabled control of system clock
I20260925 02:57:07.528216 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:33751
--webserver_interface=127.18.196.254
--webserver_port=0
--builtin_ntp_servers=127.18.196.212:40655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:33751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 02:57:07.845149 23902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:07.845482 23902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:07.845574 23902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:07.852856 23902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:07.852953 23902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:07.852989 23902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:07.853037 23902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:07.853080 23902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:07.862463 23902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:40655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:33751
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:33751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:07.863652 23902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:07.865329 23902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:07.877075 23908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:07.877115 23907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:07.877149 23910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:07.878410 23902 server_base.cc:1034] running on GCE node
I20260925 02:57:07.879202 23902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:07.880514 23902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:07.881716 23902 hybrid_clock.cc:648] HybridClock initialized: now 1790305027881667 us; error 37 us; skew 500 ppm
I20260925 02:57:07.882078 23902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:57:07.884362 23902 webserver.cc:460] Webserver started at http://127.18.196.254:34289/ using document root <none> and password file <none>
I20260925 02:57:07.885046 23902 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:07.885161 23902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:07.885499 23902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:07.888441 23902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "a5efddafcd7140c3b6a2eb12926b1c55"
format_stamp: "Formatted at 2026-09-25 02:57:07 on dist-test-slave-8zvk"
server_key: "611c121f9c95c2a363900f470929d56d"
server_key_iv: "099cef24961e1e5dad2e881db6dab568"
server_key_version: "encryptionkey@0"
I20260925 02:57:07.889639 23902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "a5efddafcd7140c3b6a2eb12926b1c55"
format_stamp: "Formatted at 2026-09-25 02:57:07 on dist-test-slave-8zvk"
server_key: "611c121f9c95c2a363900f470929d56d"
server_key_iv: "099cef24961e1e5dad2e881db6dab568"
server_key_version: "encryptionkey@0"
I20260925 02:57:07.896840 23902 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260925 02:57:07.900444 23916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:07.901659 23902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:57:07.901842 23902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "a5efddafcd7140c3b6a2eb12926b1c55"
format_stamp: "Formatted at 2026-09-25 02:57:07 on dist-test-slave-8zvk"
server_key: "611c121f9c95c2a363900f470929d56d"
server_key_iv: "099cef24961e1e5dad2e881db6dab568"
server_key_version: "encryptionkey@0"
I20260925 02:57:07.902063 23902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:07.907903 23902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:07.908638 23902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:07.908946 23902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:07.927452 23902 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:33751
I20260925 02:57:07.927490 23967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:33751 every 8 connection(s)
I20260925 02:57:07.928761 23902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:57:07.932608 23968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:07.935611 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 23902
I20260925 02:57:07.935803 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:57:07.936614 19219 external_mini_cluster.cc:1428] Setting key 4b363835b6bfe88949ba256d2303ff47
I20260925 02:57:07.945017 23968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5efddafcd7140c3b6a2eb12926b1c55: Bootstrap starting.
I20260925 02:57:07.945015 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:07.938894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44550 (local address 127.18.196.254:33751)
0925 02:57:07.939850 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:07.939860 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:07.939898 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:57:07.941498 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:07.941520 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:07.941920 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:07.942312 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:07.943426 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:07.943432 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:07.943449 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:07.943463 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:07.943515 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:07.943529 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:07.943760 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:07.943889 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 02:57:07.944138 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":340,"threads_started":1}
I20260925 02:57:07.948297 23968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5efddafcd7140c3b6a2eb12926b1c55: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:07.949275 23968 log.cc:824] T 00000000000000000000000000000000 P a5efddafcd7140c3b6a2eb12926b1c55: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:07.951869 23968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5efddafcd7140c3b6a2eb12926b1c55: No bootstrap required, opened a new log
I20260925 02:57:07.959041 23968 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5efddafcd7140c3b6a2eb12926b1c55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5efddafcd7140c3b6a2eb12926b1c55" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33751 } }
I20260925 02:57:07.959375 23968 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5efddafcd7140c3b6a2eb12926b1c55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:07.959488 23968 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5efddafcd7140c3b6a2eb12926b1c55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5efddafcd7140c3b6a2eb12926b1c55, State: Initialized, Role: FOLLOWER
I20260925 02:57:07.960100 23968 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5efddafcd7140c3b6a2eb12926b1c55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5efddafcd7140c3b6a2eb12926b1c55" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33751 } }
I20260925 02:57:07.960326 23968 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5efddafcd7140c3b6a2eb12926b1c55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:57:07.960489 23968 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5efddafcd7140c3b6a2eb12926b1c55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:57:07.960664 23968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5efddafcd7140c3b6a2eb12926b1c55 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:07.962788 23968 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5efddafcd7140c3b6a2eb12926b1c55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5efddafcd7140c3b6a2eb12926b1c55" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33751 } }
I20260925 02:57:07.963330 23968 leader_election.cc:302] T 00000000000000000000000000000000 P a5efddafcd7140c3b6a2eb12926b1c55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5efddafcd7140c3b6a2eb12926b1c55; no voters:
I20260925 02:57:07.963884 23968 leader_election.cc:288] T 00000000000000000000000000000000 P a5efddafcd7140c3b6a2eb12926b1c55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:57:07.964027 23973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5efddafcd7140c3b6a2eb12926b1c55 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:07.964743 23973 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5efddafcd7140c3b6a2eb12926b1c55 [term 1 LEADER]: Becoming Leader. State: Replica: a5efddafcd7140c3b6a2eb12926b1c55, State: Running, Role: LEADER
I20260925 02:57:07.965288 23973 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5efddafcd7140c3b6a2eb12926b1c55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5efddafcd7140c3b6a2eb12926b1c55" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33751 } }
I20260925 02:57:07.965873 23968 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5efddafcd7140c3b6a2eb12926b1c55 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:07.968014 23974 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5efddafcd7140c3b6a2eb12926b1c55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5efddafcd7140c3b6a2eb12926b1c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5efddafcd7140c3b6a2eb12926b1c55" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33751 } } }
I20260925 02:57:07.968482 23974 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5efddafcd7140c3b6a2eb12926b1c55 [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:07.968333 23975 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5efddafcd7140c3b6a2eb12926b1c55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5efddafcd7140c3b6a2eb12926b1c55. Latest consensus state: current_term: 1 leader_uuid: "a5efddafcd7140c3b6a2eb12926b1c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5efddafcd7140c3b6a2eb12926b1c55" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33751 } } }
I20260925 02:57:07.968676 23975 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5efddafcd7140c3b6a2eb12926b1c55 [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:07.970889 23982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:07.975664 23982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:07.984028 23982 catalog_manager.cc:1347] Generated new cluster ID: efdaf68cb3734313b63fe0cbd5eec0e7
I20260925 02:57:07.984154 23982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:08.006856 23982 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:57:08.007292 23982 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:08.013998 23982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5efddafcd7140c3b6a2eb12926b1c55: Generated new TSK 0
I20260925 02:57:08.014761 23982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:57:08.030205 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33751
--builtin_ntp_servers=127.18.196.212:40655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 02:57:08.346534 23992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:08.346818 23992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:08.347018 23992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:08.353984 23992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:08.354103 23992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:08.354300 23992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 02:57:08.363068 23992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:40655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:08.364287 23992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:08.366094 23992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:08.379339 23997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:08.379247 23998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:08.379297 24000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:08.380443 23992 server_base.cc:1034] running on GCE node
I20260925 02:57:08.381042 23992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:08.382023 23992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:08.383248 23992 hybrid_clock.cc:648] HybridClock initialized: now 1790305028383193 us; error 44 us; skew 500 ppm
I20260925 02:57:08.383565 23992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:57:08.385670 23992 webserver.cc:460] Webserver started at http://127.18.196.193:37741/ using document root <none> and password file <none>
I20260925 02:57:08.386339 23992 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:08.386475 23992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:08.386790 23992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:08.389546 23992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "d382fd8ba89b4c0e876467508a5b7e84"
format_stamp: "Formatted at 2026-09-25 02:57:08 on dist-test-slave-8zvk"
server_key: "3d00d01d35654ac75e157260ce0bf738"
server_key_iv: "246b384989a706e735203e2e669f8879"
server_key_version: "encryptionkey@0"
I20260925 02:57:08.390334 23992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "d382fd8ba89b4c0e876467508a5b7e84"
format_stamp: "Formatted at 2026-09-25 02:57:08 on dist-test-slave-8zvk"
server_key: "3d00d01d35654ac75e157260ce0bf738"
server_key_iv: "246b384989a706e735203e2e669f8879"
server_key_version: "encryptionkey@0"
I20260925 02:57:08.395409 23992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 02:57:08.398789 24006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:08.400172 23992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:57:08.400367 23992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "d382fd8ba89b4c0e876467508a5b7e84"
format_stamp: "Formatted at 2026-09-25 02:57:08 on dist-test-slave-8zvk"
server_key: "3d00d01d35654ac75e157260ce0bf738"
server_key_iv: "246b384989a706e735203e2e669f8879"
server_key_version: "encryptionkey@0"
I20260925 02:57:08.400600 23992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:08.406446 23992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:08.407255 23992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:08.407541 23992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:08.408427 23992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:08.409813 23992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:08.409933 23992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:08.410066 23992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:08.410153 23992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:08.444180 23992 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:42751
I20260925 02:57:08.444190 24120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:42751 every 8 connection(s)
I20260925 02:57:08.445732 23992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 02:57:08.453608 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 23992
I20260925 02:57:08.453902 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 02:57:08.454519 24123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:08.448448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:46215 (local address 127.18.196.254:33751)
0925 02:57:08.448876 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:08.448881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:08.450063 (+ 1182us) server_negotiation.cc:407] Connection header received
0925 02:57:08.452027 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:08.452033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:08.452103 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:08.452235 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:08.453775 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:08.453780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:08.453784 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:08.453800 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:08.453848 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:08.453866 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:08.454032 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:08.454257 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 02:57:08.454307 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":202,"threads_started":1}
I20260925 02:57:08.454835 19219 external_mini_cluster.cc:1428] Setting key 172afa371f4f60ed743f584ae421dd12
I20260925 02:57:08.455907 24124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:08.449010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:33751 (local address 127.18.196.193:46215)
0925 02:57:08.449910 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:08.449945 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:08.451761 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:08.452509 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:08.452518 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:08.452607 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:08.453328 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:57:08.453333 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:08.453413 (+ 80us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:57:08.453538 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:08.454014 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:08.454027 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:08.454036 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 02:57:08.454359 (+ 323us) client_negotiation.cc:240] Negotiation successful
0925 02:57:08.454646 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":277,"threads_started":1}
I20260925 02:57:08.457559 24121 heartbeater.cc:347] Connected to a master server at 127.18.196.254:33751
I20260925 02:57:08.457867 24121 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:08.458099 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33751
--builtin_ntp_servers=127.18.196.212:40655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 02:57:08.458571 24121 heartbeater.cc:511] Master 127.18.196.254:33751 requested a full tablet report, sending...
I20260925 02:57:08.460691 23932 ts_manager.cc:194] Registered new tserver with Master: d382fd8ba89b4c0e876467508a5b7e84 (127.18.196.193:42751)
W20260925 02:57:08.767694 24125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:08.767939 24125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:08.768021 24125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:08.775808 24125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:08.775902 24125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:08.776005 24125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 02:57:08.784862 24125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:40655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:08.786098 24125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:08.787717 24125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:08.800045 24131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:08.800084 24130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:08.800598 24133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:08.800693 24125 server_base.cc:1034] running on GCE node
I20260925 02:57:08.801280 24125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:08.802140 24125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:08.803334 24125 hybrid_clock.cc:648] HybridClock initialized: now 1790305028803301 us; error 31 us; skew 500 ppm
I20260925 02:57:08.803607 24125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:57:08.806063 24125 webserver.cc:460] Webserver started at http://127.18.196.194:42959/ using document root <none> and password file <none>
I20260925 02:57:08.806732 24125 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:08.806818 24125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:08.807196 24125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:08.809813 24125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "049499bbe940444e81225f8a701441ec"
format_stamp: "Formatted at 2026-09-25 02:57:08 on dist-test-slave-8zvk"
server_key: "6bb4d1fcbf95e39491d60c19bdfce758"
server_key_iv: "6b77829dd041eff139a0e048916712e4"
server_key_version: "encryptionkey@0"
I20260925 02:57:08.810587 24125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "049499bbe940444e81225f8a701441ec"
format_stamp: "Formatted at 2026-09-25 02:57:08 on dist-test-slave-8zvk"
server_key: "6bb4d1fcbf95e39491d60c19bdfce758"
server_key_iv: "6b77829dd041eff139a0e048916712e4"
server_key_version: "encryptionkey@0"
I20260925 02:57:08.815433 24125 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 02:57:08.818499 24139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:08.819664 24125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:57:08.819833 24125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "049499bbe940444e81225f8a701441ec"
format_stamp: "Formatted at 2026-09-25 02:57:08 on dist-test-slave-8zvk"
server_key: "6bb4d1fcbf95e39491d60c19bdfce758"
server_key_iv: "6b77829dd041eff139a0e048916712e4"
server_key_version: "encryptionkey@0"
I20260925 02:57:08.820070 24125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:08.825224 24125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:08.825950 24125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:08.826203 24125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:08.827117 24125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:08.828406 24125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:08.828534 24125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:08.828658 24125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:08.828743 24125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:08.860109 24125 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:37925
I20260925 02:57:08.860165 24251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:37925 every 8 connection(s)
I20260925 02:57:08.861439 24125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 02:57:08.866473 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 24125
I20260925 02:57:08.866640 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 02:57:08.867183 19219 external_mini_cluster.cc:1428] Setting key 419efbd695bfc9bebbfc263397d6cd72
I20260925 02:57:08.869755 24123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:08.863926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:54933 (local address 127.18.196.254:33751)
0925 02:57:08.864146 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:08.864152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:08.865445 (+ 1293us) server_negotiation.cc:407] Connection header received
0925 02:57:08.867458 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:08.867465 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:08.867534 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:08.867647 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:08.868999 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:08.869005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:08.869010 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:08.869029 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:08.869068 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:08.869080 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:08.869198 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:08.869470 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 02:57:08.869531 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 02:57:08.870606 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33751
--builtin_ntp_servers=127.18.196.212:40655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 02:57:08.870959 24254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:08.864438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:33751 (local address 127.18.196.194:54933)
0925 02:57:08.865287 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:08.865325 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:08.867123 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:08.867832 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:08.867840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:08.867921 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:08.868633 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:57:08.868638 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:08.868684 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:57:08.868783 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:08.869188 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:08.869199 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:08.869207 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 02:57:08.869521 (+ 314us) client_negotiation.cc:240] Negotiation successful
0925 02:57:08.869761 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":255,"threads_started":1}
I20260925 02:57:08.872257 24252 heartbeater.cc:347] Connected to a master server at 127.18.196.254:33751
I20260925 02:57:08.872575 24252 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:08.873203 24252 heartbeater.cc:511] Master 127.18.196.254:33751 requested a full tablet report, sending...
I20260925 02:57:08.874696 23932 ts_manager.cc:194] Registered new tserver with Master: 049499bbe940444e81225f8a701441ec (127.18.196.194:37925)
W20260925 02:57:09.188740 24255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:09.189040 24255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:09.189216 24255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:09.196170 24255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:09.196292 24255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:09.196451 24255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 02:57:09.205009 24255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:40655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:09.206230 24255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:09.207988 24255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:09.223215 24260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:09.223373 24264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:09.223320 24261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:09.224388 24255 server_base.cc:1034] running on GCE node
I20260925 02:57:09.225001 24255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:09.225927 24255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:09.227144 24255 hybrid_clock.cc:648] HybridClock initialized: now 1790305029227106 us; error 47 us; skew 500 ppm
I20260925 02:57:09.227453 24255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:57:09.229470 24255 webserver.cc:460] Webserver started at http://127.18.196.195:45761/ using document root <none> and password file <none>
I20260925 02:57:09.230098 24255 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:09.230225 24255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:09.230543 24255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:09.233259 24255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "3ddd2b9c7b714b65bc5d9b20d8d520f7"
format_stamp: "Formatted at 2026-09-25 02:57:09 on dist-test-slave-8zvk"
server_key: "e874bc324ede155fcb23d29b206691ae"
server_key_iv: "33d6cdfab8435b6f427fda6b814cf1b3"
server_key_version: "encryptionkey@0"
I20260925 02:57:09.234027 24255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "3ddd2b9c7b714b65bc5d9b20d8d520f7"
format_stamp: "Formatted at 2026-09-25 02:57:09 on dist-test-slave-8zvk"
server_key: "e874bc324ede155fcb23d29b206691ae"
server_key_iv: "33d6cdfab8435b6f427fda6b814cf1b3"
server_key_version: "encryptionkey@0"
I20260925 02:57:09.238946 24255 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260925 02:57:09.242115 24270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:09.243350 24255 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:57:09.243515 24255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "3ddd2b9c7b714b65bc5d9b20d8d520f7"
format_stamp: "Formatted at 2026-09-25 02:57:09 on dist-test-slave-8zvk"
server_key: "e874bc324ede155fcb23d29b206691ae"
server_key_iv: "33d6cdfab8435b6f427fda6b814cf1b3"
server_key_version: "encryptionkey@0"
I20260925 02:57:09.243723 24255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:09.249433 24255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:09.250155 24255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:09.250409 24255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:09.251282 24255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:09.252591 24255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:09.252704 24255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:09.252825 24255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:09.252908 24255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:09.284869 24255 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:45357
I20260925 02:57:09.284953 24383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:45357 every 8 connection(s)
I20260925 02:57:09.286211 24255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 02:57:09.290195 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 24255
I20260925 02:57:09.290382 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 02:57:09.290867 19219 external_mini_cluster.cc:1428] Setting key c25e961864f43f75e109f8b10a4cbb84
I20260925 02:57:09.294322 24123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.288654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:52775 (local address 127.18.196.254:33751)
0925 02:57:09.288840 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:09.288844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:09.290134 (+ 1290us) server_negotiation.cc:407] Connection header received
0925 02:57:09.292205 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:09.292210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:09.292279 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:09.292414 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:09.293703 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:09.293708 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:09.293718 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:09.293733 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:09.293763 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:09.293774 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:09.293915 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:09.294077 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 02:57:09.294123 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:57:09.295193 24386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.289128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:33751 (local address 127.18.196.195:52775)
0925 02:57:09.289997 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:09.290029 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:09.291808 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:09.292622 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:09.292635 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:09.292747 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:09.293413 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:57:09.293415 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:09.293450 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:57:09.293527 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:09.293892 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:09.293900 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:09.293905 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:57:09.294106 (+ 201us) client_negotiation.cc:240] Negotiation successful
0925 02:57:09.294272 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":282,"threads_started":1}
I20260925 02:57:09.296386 24384 heartbeater.cc:347] Connected to a master server at 127.18.196.254:33751
I20260925 02:57:09.296635 24384 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:09.297152 24384 heartbeater.cc:511] Master 127.18.196.254:33751 requested a full tablet report, sending...
I20260925 02:57:09.298442 23932 ts_manager.cc:194] Registered new tserver with Master: 3ddd2b9c7b714b65bc5d9b20d8d520f7 (127.18.196.195:45357)
I20260925 02:57:09.305968 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:57:09.313009 24123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.310325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44556 (local address 127.18.196.254:33751)
0925 02:57:09.310562 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:09.310571 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:09.311045 (+ 474us) server_negotiation.cc:407] Connection header received
0925 02:57:09.311236 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:09.311241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:09.311297 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:09.311397 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:09.312376 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:09.312382 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:09.312386 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:09.312403 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:09.312443 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:09.312453 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:09.312573 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:09.312662 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 02:57:09.312720 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 02:57:09.316614 23932 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:57:09.322991 23932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:57:09.341586 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.333960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:45357 (local address 127.0.0.1:58946)
0925 02:57:09.336275 (+ 2315us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:09.336292 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:09.336373 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:09.338454 (+ 2081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:09.338464 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:09.338565 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:09.339487 (+ 922us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:57:09.339493 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:09.339532 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:57:09.339629 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:09.340636 (+ 1007us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:09.340651 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:09.340662 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 02:57:09.340859 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 02:57:09.340957 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2199,"thread_start_us":1596,"threads_started":1}
I20260925 02:57:09.342620 24399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.334594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58946 (local address 127.18.196.195:45357)
0925 02:57:09.336642 (+ 2048us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:09.336650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:09.336683 (+ 33us) server_negotiation.cc:407] Connection header received
0925 02:57:09.336750 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:09.336756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:09.336955 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:09.337125 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:09.340208 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:09.340220 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:09.340239 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:09.340266 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:09.340339 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:09.340378 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:09.340569 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:09.341975 (+ 1406us) server_negotiation.cc:299] Negotiation successful
0925 02:57:09.342094 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1884,"thread_start_us":2060,"threads_started":1}
I20260925 02:57:09.342878 24398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.333950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55728 (local address 127.18.196.193:42751)
0925 02:57:09.335394 (+ 1444us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:09.335402 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:09.336125 (+ 723us) server_negotiation.cc:407] Connection header received
0925 02:57:09.336425 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:09.336433 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:09.336626 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:09.336770 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:09.340382 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:09.340395 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:09.340414 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:09.340449 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:09.340519 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:09.340543 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:09.341946 (+ 1403us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:09.342265 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 02:57:09.342404 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1254,"thread_start_us":1139,"threads_started":1}
I20260925 02:57:09.343864 24397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.333800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:42751 (local address 127.0.0.1:55728)
0925 02:57:09.335950 (+ 2150us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:09.336007 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:09.336233 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:09.339171 (+ 2938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:09.339178 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:09.339211 (+ 33us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:09.340024 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:57:09.340030 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:09.340078 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:57:09.340149 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:09.342073 (+ 1924us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:09.342081 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:09.342088 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 02:57:09.343200 (+ 1112us) client_negotiation.cc:240] Negotiation successful
0925 02:57:09.343264 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1884,"thread_start_us":2193,"threads_started":1}
I20260925 02:57:09.347460 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.336408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:37925 (local address 127.0.0.1:47952)
0925 02:57:09.343263 (+ 6855us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:09.343285 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:09.343391 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:09.345055 (+ 1664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:09.345059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:09.345095 (+ 36us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:09.345646 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:57:09.345650 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:09.345667 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:57:09.345713 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:09.347066 (+ 1353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:09.347071 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:09.347076 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:57:09.347172 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 02:57:09.347218 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6717,"thread_start_us":7397,"threads_started":1}
I20260925 02:57:09.347816 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.336971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47952 (local address 127.18.196.194:37925)
0925 02:57:09.344567 (+ 7596us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:09.344575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:09.344600 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:57:09.344667 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:09.344675 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:09.344859 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:09.345876 (+ 1017us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:09.346803 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:09.346811 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:09.346823 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:09.346874 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:09.346929 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:09.346950 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:09.347166 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:09.347240 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 02:57:09.347322 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7440,"thread_start_us":7871,"threads_started":1}
I20260925 02:57:09.347460 24055 tablet_service.cc:1467] Processing CreateTablet for tablet 8b7df659b1e243d7ab592427220a4fc5 (DEFAULT_TABLE table=test-table [id=67e214fee64949328eed2a265ef00f7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:57:09.347563 24319 tablet_service.cc:1467] Processing CreateTablet for tablet 8b7df659b1e243d7ab592427220a4fc5 (DEFAULT_TABLE table=test-table [id=67e214fee64949328eed2a265ef00f7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:57:09.348788 24055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b7df659b1e243d7ab592427220a4fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:09.348959 24319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b7df659b1e243d7ab592427220a4fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:09.351274 24187 tablet_service.cc:1467] Processing CreateTablet for tablet 8b7df659b1e243d7ab592427220a4fc5 (DEFAULT_TABLE table=test-table [id=67e214fee64949328eed2a265ef00f7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:57:09.352495 24187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b7df659b1e243d7ab592427220a4fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:09.359611 24403 tablet_bootstrap.cc:492] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7: Bootstrap starting.
I20260925 02:57:09.362573 24403 tablet_bootstrap.cc:654] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:09.362872 24404 tablet_bootstrap.cc:492] T 8b7df659b1e243d7ab592427220a4fc5 P d382fd8ba89b4c0e876467508a5b7e84: Bootstrap starting.
I20260925 02:57:09.364804 24403 log.cc:824] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:09.365648 24405 tablet_bootstrap.cc:492] T 8b7df659b1e243d7ab592427220a4fc5 P 049499bbe940444e81225f8a701441ec: Bootstrap starting.
I20260925 02:57:09.366504 24404 tablet_bootstrap.cc:654] T 8b7df659b1e243d7ab592427220a4fc5 P d382fd8ba89b4c0e876467508a5b7e84: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:09.367626 24404 log.cc:824] T 8b7df659b1e243d7ab592427220a4fc5 P d382fd8ba89b4c0e876467508a5b7e84: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:09.368916 24405 tablet_bootstrap.cc:654] T 8b7df659b1e243d7ab592427220a4fc5 P 049499bbe940444e81225f8a701441ec: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:09.369593 24403 tablet_bootstrap.cc:492] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7: No bootstrap required, opened a new log
I20260925 02:57:09.369863 24403 ts_tablet_manager.cc:1397] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 02:57:09.370046 24405 log.cc:824] T 8b7df659b1e243d7ab592427220a4fc5 P 049499bbe940444e81225f8a701441ec: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:09.370115 24404 tablet_bootstrap.cc:492] T 8b7df659b1e243d7ab592427220a4fc5 P d382fd8ba89b4c0e876467508a5b7e84: No bootstrap required, opened a new log
I20260925 02:57:09.370342 24404 ts_tablet_manager.cc:1397] T 8b7df659b1e243d7ab592427220a4fc5 P d382fd8ba89b4c0e876467508a5b7e84: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 02:57:09.372773 24405 tablet_bootstrap.cc:492] T 8b7df659b1e243d7ab592427220a4fc5 P 049499bbe940444e81225f8a701441ec: No bootstrap required, opened a new log
I20260925 02:57:09.373009 24405 ts_tablet_manager.cc:1397] T 8b7df659b1e243d7ab592427220a4fc5 P 049499bbe940444e81225f8a701441ec: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260925 02:57:09.377825 24403 raft_consensus.cc:348] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d382fd8ba89b4c0e876467508a5b7e84" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42751 } } peers { permanent_uuid: "3ddd2b9c7b714b65bc5d9b20d8d520f7" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45357 } } peers { permanent_uuid: "049499bbe940444e81225f8a701441ec" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37925 } }
I20260925 02:57:09.378178 24403 raft_consensus.cc:374] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:09.378280 24403 raft_consensus.cc:729] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ddd2b9c7b714b65bc5d9b20d8d520f7, State: Initialized, Role: FOLLOWER
I20260925 02:57:09.378914 24403 consensus_queue.cc:260] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d382fd8ba89b4c0e876467508a5b7e84" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42751 } } peers { permanent_uuid: "3ddd2b9c7b714b65bc5d9b20d8d520f7" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45357 } } peers { permanent_uuid: "049499bbe940444e81225f8a701441ec" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37925 } }
I20260925 02:57:09.379978 24384 heartbeater.cc:503] Master 127.18.196.254:33751 was elected leader, sending a full tablet report...
I20260925 02:57:09.380589 24403 ts_tablet_manager.cc:1428] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260925 02:57:09.380576 24404 raft_consensus.cc:348] T 8b7df659b1e243d7ab592427220a4fc5 P d382fd8ba89b4c0e876467508a5b7e84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d382fd8ba89b4c0e876467508a5b7e84" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42751 } } peers { permanent_uuid: "3ddd2b9c7b714b65bc5d9b20d8d520f7" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45357 } } peers { permanent_uuid: "049499bbe940444e81225f8a701441ec" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37925 } }
I20260925 02:57:09.380959 24404 raft_consensus.cc:374] T 8b7df659b1e243d7ab592427220a4fc5 P d382fd8ba89b4c0e876467508a5b7e84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:09.381044 24404 raft_consensus.cc:729] T 8b7df659b1e243d7ab592427220a4fc5 P d382fd8ba89b4c0e876467508a5b7e84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d382fd8ba89b4c0e876467508a5b7e84, State: Initialized, Role: FOLLOWER
I20260925 02:57:09.381703 24404 consensus_queue.cc:260] T 8b7df659b1e243d7ab592427220a4fc5 P d382fd8ba89b4c0e876467508a5b7e84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d382fd8ba89b4c0e876467508a5b7e84" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42751 } } peers { permanent_uuid: "3ddd2b9c7b714b65bc5d9b20d8d520f7" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45357 } } peers { permanent_uuid: "049499bbe940444e81225f8a701441ec" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37925 } }
I20260925 02:57:09.382750 24121 heartbeater.cc:503] Master 127.18.196.254:33751 was elected leader, sending a full tablet report...
I20260925 02:57:09.383072 24405 raft_consensus.cc:348] T 8b7df659b1e243d7ab592427220a4fc5 P 049499bbe940444e81225f8a701441ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d382fd8ba89b4c0e876467508a5b7e84" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42751 } } peers { permanent_uuid: "3ddd2b9c7b714b65bc5d9b20d8d520f7" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45357 } } peers { permanent_uuid: "049499bbe940444e81225f8a701441ec" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37925 } }
I20260925 02:57:09.383370 24404 ts_tablet_manager.cc:1428] T 8b7df659b1e243d7ab592427220a4fc5 P d382fd8ba89b4c0e876467508a5b7e84: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260925 02:57:09.383483 24405 raft_consensus.cc:374] T 8b7df659b1e243d7ab592427220a4fc5 P 049499bbe940444e81225f8a701441ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:09.383566 24405 raft_consensus.cc:729] T 8b7df659b1e243d7ab592427220a4fc5 P 049499bbe940444e81225f8a701441ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 049499bbe940444e81225f8a701441ec, State: Initialized, Role: FOLLOWER
I20260925 02:57:09.384173 24405 consensus_queue.cc:260] T 8b7df659b1e243d7ab592427220a4fc5 P 049499bbe940444e81225f8a701441ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d382fd8ba89b4c0e876467508a5b7e84" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42751 } } peers { permanent_uuid: "3ddd2b9c7b714b65bc5d9b20d8d520f7" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45357 } } peers { permanent_uuid: "049499bbe940444e81225f8a701441ec" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37925 } }
I20260925 02:57:09.385360 24252 heartbeater.cc:503] Master 127.18.196.254:33751 was elected leader, sending a full tablet report...
I20260925 02:57:09.386055 24405 ts_tablet_manager.cc:1428] T 8b7df659b1e243d7ab592427220a4fc5 P 049499bbe940444e81225f8a701441ec: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260925 02:57:09.448022 24122 tablet.cc:2369] T 8b7df659b1e243d7ab592427220a4fc5 P d382fd8ba89b4c0e876467508a5b7e84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:57:09.465370 24409 raft_consensus.cc:482] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:57:09.465621 24409 raft_consensus.cc:504] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d382fd8ba89b4c0e876467508a5b7e84" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42751 } } peers { permanent_uuid: "3ddd2b9c7b714b65bc5d9b20d8d520f7" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45357 } } peers { permanent_uuid: "049499bbe940444e81225f8a701441ec" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37925 } }
I20260925 02:57:09.467343 24409 leader_election.cc:288] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d382fd8ba89b4c0e876467508a5b7e84 (127.18.196.193:42751), 049499bbe940444e81225f8a701441ec (127.18.196.194:37925)
I20260925 02:57:09.470242 24386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.467812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:37925 (local address 127.18.196.195:43331)
0925 02:57:09.468017 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:09.468036 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:09.468154 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:09.468556 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:09.468560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:09.468590 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:09.469258 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:57:09.469265 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:09.469288 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:57:09.469398 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:09.469779 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:09.469784 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:09.469789 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:57:09.469934 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 02:57:09.469985 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 02:57:09.470255 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.467784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:43331 (local address 127.18.196.194:37925)
0925 02:57:09.467963 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:09.467971 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:09.468137 (+ 166us) server_negotiation.cc:407] Connection header received
0925 02:57:09.468329 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:09.468334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:09.468391 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:09.468512 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:09.469559 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:09.469566 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:09.469571 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:09.469591 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:09.469636 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:09.469653 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:09.469787 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:09.469929 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 02:57:09.469992 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:57:09.471024 24398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.467989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:46291 (local address 127.18.196.193:42751)
0925 02:57:09.468169 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:09.468176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:09.468856 (+ 680us) server_negotiation.cc:407] Connection header received
0925 02:57:09.469031 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:09.469035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:09.469080 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:09.469185 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:09.470172 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:09.470177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:09.470181 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:09.470198 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:09.470235 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:09.470254 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:09.470370 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:09.470704 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 02:57:09.470766 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:57:09.471101 24412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.467811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:42751 (local address 127.18.196.195:46291)
0925 02:57:09.468756 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:09.468769 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:09.468899 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:09.469206 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:09.469211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:09.469255 (+ 44us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:09.469954 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:57:09.469957 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:09.469973 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:57:09.470014 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:09.470571 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:09.470574 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:09.470577 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:57:09.470665 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 02:57:09.470700 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"mutex_wait_us":179,"thread_start_us":437,"threads_started":1}
I20260925 02:57:09.471330 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7df659b1e243d7ab592427220a4fc5" candidate_uuid: "3ddd2b9c7b714b65bc5d9b20d8d520f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "049499bbe940444e81225f8a701441ec" is_pre_election: true
I20260925 02:57:09.471805 24207 raft_consensus.cc:2454] T 8b7df659b1e243d7ab592427220a4fc5 P 049499bbe940444e81225f8a701441ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ddd2b9c7b714b65bc5d9b20d8d520f7 in term 0.
I20260925 02:57:09.471997 24075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7df659b1e243d7ab592427220a4fc5" candidate_uuid: "3ddd2b9c7b714b65bc5d9b20d8d520f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d382fd8ba89b4c0e876467508a5b7e84" is_pre_election: true
I20260925 02:57:09.472463 24273 leader_election.cc:302] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 049499bbe940444e81225f8a701441ec, 3ddd2b9c7b714b65bc5d9b20d8d520f7; no voters:
I20260925 02:57:09.472565 24075 raft_consensus.cc:2454] T 8b7df659b1e243d7ab592427220a4fc5 P d382fd8ba89b4c0e876467508a5b7e84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ddd2b9c7b714b65bc5d9b20d8d520f7 in term 0.
I20260925 02:57:09.472867 24409 raft_consensus.cc:2781] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:57:09.472976 24409 raft_consensus.cc:482] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:57:09.473099 24409 raft_consensus.cc:3037] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:09.475355 24409 raft_consensus.cc:504] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d382fd8ba89b4c0e876467508a5b7e84" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42751 } } peers { permanent_uuid: "3ddd2b9c7b714b65bc5d9b20d8d520f7" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45357 } } peers { permanent_uuid: "049499bbe940444e81225f8a701441ec" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37925 } }
I20260925 02:57:09.475917 24409 leader_election.cc:288] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7 [CANDIDATE]: Term 1 election: Requested vote from peers d382fd8ba89b4c0e876467508a5b7e84 (127.18.196.193:42751), 049499bbe940444e81225f8a701441ec (127.18.196.194:37925)
I20260925 02:57:09.476264 24075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7df659b1e243d7ab592427220a4fc5" candidate_uuid: "3ddd2b9c7b714b65bc5d9b20d8d520f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d382fd8ba89b4c0e876467508a5b7e84"
I20260925 02:57:09.476473 24075 raft_consensus.cc:3037] T 8b7df659b1e243d7ab592427220a4fc5 P d382fd8ba89b4c0e876467508a5b7e84 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:09.476437 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7df659b1e243d7ab592427220a4fc5" candidate_uuid: "3ddd2b9c7b714b65bc5d9b20d8d520f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "049499bbe940444e81225f8a701441ec"
I20260925 02:57:09.476625 24207 raft_consensus.cc:3037] T 8b7df659b1e243d7ab592427220a4fc5 P 049499bbe940444e81225f8a701441ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:09.478588 24075 raft_consensus.cc:2454] T 8b7df659b1e243d7ab592427220a4fc5 P d382fd8ba89b4c0e876467508a5b7e84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ddd2b9c7b714b65bc5d9b20d8d520f7 in term 1.
I20260925 02:57:09.478880 24207 raft_consensus.cc:2454] T 8b7df659b1e243d7ab592427220a4fc5 P 049499bbe940444e81225f8a701441ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ddd2b9c7b714b65bc5d9b20d8d520f7 in term 1.
I20260925 02:57:09.479014 24272 leader_election.cc:302] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ddd2b9c7b714b65bc5d9b20d8d520f7, d382fd8ba89b4c0e876467508a5b7e84; no voters:
I20260925 02:57:09.479324 24409 raft_consensus.cc:2781] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:09.479748 24409 raft_consensus.cc:686] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7 [term 1 LEADER]: Becoming Leader. State: Replica: 3ddd2b9c7b714b65bc5d9b20d8d520f7, State: Running, Role: LEADER
I20260925 02:57:09.480247 24409 consensus_queue.cc:237] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d382fd8ba89b4c0e876467508a5b7e84" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42751 } } peers { permanent_uuid: "3ddd2b9c7b714b65bc5d9b20d8d520f7" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45357 } } peers { permanent_uuid: "049499bbe940444e81225f8a701441ec" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37925 } }
I20260925 02:57:09.484635 23932 catalog_manager.cc:5495] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ddd2b9c7b714b65bc5d9b20d8d520f7 (127.18.196.195). New cstate: current_term: 1 leader_uuid: "3ddd2b9c7b714b65bc5d9b20d8d520f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d382fd8ba89b4c0e876467508a5b7e84" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ddd2b9c7b714b65bc5d9b20d8d520f7" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "049499bbe940444e81225f8a701441ec" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37925 } health_report { overall_health: UNKNOWN } } }
I20260925 02:57:09.532457 24399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.530572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58960 (local address 127.18.196.195:45357)
0925 02:57:09.530741 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:09.530746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:09.530871 (+ 125us) server_negotiation.cc:407] Connection header received
0925 02:57:09.531047 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:09.531051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:09.531142 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:09.531234 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:09.532002 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:09.532007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:09.532010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:09.532024 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:09.532056 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:09.532068 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:09.532166 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:09.532223 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 02:57:09.532266 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 02:57:09.537822 24385 tablet.cc:2369] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:57:09.540616 24207 raft_consensus.cc:1261] T 8b7df659b1e243d7ab592427220a4fc5 P 049499bbe940444e81225f8a701441ec [term 1 FOLLOWER]: Refusing update from remote peer 3ddd2b9c7b714b65bc5d9b20d8d520f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:57:09.540616 24075 raft_consensus.cc:1261] T 8b7df659b1e243d7ab592427220a4fc5 P d382fd8ba89b4c0e876467508a5b7e84 [term 1 FOLLOWER]: Refusing update from remote peer 3ddd2b9c7b714b65bc5d9b20d8d520f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:57:09.541574 24413 consensus_queue.cc:1035] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d382fd8ba89b4c0e876467508a5b7e84" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:57:09.541905 24409 consensus_queue.cc:1035] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "049499bbe940444e81225f8a701441ec" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:57:09.552609 24418 mvcc.cc:205] Tried to move back new op lower bound from 7333089400988041216 to 7333089400754835456. Current Snapshot: MvccSnapshot[applied={T|T < 7333089400988041216}]
I20260925 02:57:09.553404 24416 mvcc.cc:205] Tried to move back new op lower bound from 7333089400988041216 to 7333089400754835456. Current Snapshot: MvccSnapshot[applied={T|T < 7333089400988041216}]
I20260925 02:57:09.565727 24417 mvcc.cc:205] Tried to move back new op lower bound from 7333089400988041216 to 7333089400754835456. Current Snapshot: MvccSnapshot[applied={T|T < 7333089400988041216}]
I20260925 02:57:09.569227 23932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:57:09.569512 23932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:57:09.573032 23932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5efddafcd7140c3b6a2eb12926b1c55: Sending DeleteTablet for 3 replicas of tablet 8b7df659b1e243d7ab592427220a4fc5
I20260925 02:57:09.574210 24055 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b7df659b1e243d7ab592427220a4fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:09 UTC) from {username='slave'} at 127.0.0.1:55728
I20260925 02:57:09.574258 24319 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b7df659b1e243d7ab592427220a4fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:09 UTC) from {username='slave'} at 127.0.0.1:58946
I20260925 02:57:09.574697 24187 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b7df659b1e243d7ab592427220a4fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:09 UTC) from {username='slave'} at 127.0.0.1:47952
I20260925 02:57:09.576957 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 23992
I20260925 02:57:09.590502 24428 tablet_replica.cc:330] T 8b7df659b1e243d7ab592427220a4fc5 P 049499bbe940444e81225f8a701441ec: stopping tablet replica
I20260925 02:57:09.591143 24428 raft_consensus.cc:2229] T 8b7df659b1e243d7ab592427220a4fc5 P 049499bbe940444e81225f8a701441ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:57:09.591821 24428 raft_consensus.cc:2258] T 8b7df659b1e243d7ab592427220a4fc5 P 049499bbe940444e81225f8a701441ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:57:09.595937 24398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.578419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55732 (local address 127.18.196.193:42751)
0925 02:57:09.578656 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:09.578666 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:09.592819 (+ 14153us) server_negotiation.cc:407] Connection header received
0925 02:57:09.593048 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:09.593057 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:09.593209 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:09.593371 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:09.594809 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:09.594819 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:09.594827 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:09.594892 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:09.594947 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:09.594970 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:09.595099 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:09.595397 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 02:57:09.595481 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 02:57:09.597996 24116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:09.601168 24428 ts_tablet_manager.cc:1905] T 8b7df659b1e243d7ab592427220a4fc5 P 049499bbe940444e81225f8a701441ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:57:09.605360 24426 tablet_replica.cc:330] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7: stopping tablet replica
I20260925 02:57:09.605948 24426 raft_consensus.cc:2229] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:57:09.606753 24426 raft_consensus.cc:2258] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:57:09.612823 24426 ts_tablet_manager.cc:1905] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:57:09.614665 24428 ts_tablet_manager.cc:1918] T 8b7df659b1e243d7ab592427220a4fc5 P 049499bbe940444e81225f8a701441ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:57:09.614848 24428 log.cc:1196] T 8b7df659b1e243d7ab592427220a4fc5 P 049499bbe940444e81225f8a701441ec: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/8b7df659b1e243d7ab592427220a4fc5
I20260925 02:57:09.615370 24428 ts_tablet_manager.cc:1939] T 8b7df659b1e243d7ab592427220a4fc5 P 049499bbe940444e81225f8a701441ec: Deleting consensus metadata
I20260925 02:57:09.617297 23919 catalog_manager.cc:4841] TS 049499bbe940444e81225f8a701441ec (127.18.196.194:37925): tablet 8b7df659b1e243d7ab592427220a4fc5 (table test-table [id=67e214fee64949328eed2a265ef00f7d]) successfully deleted
I20260925 02:57:09.619005 24426 ts_tablet_manager.cc:1918] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:57:09.619169 24426 log.cc:1196] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/8b7df659b1e243d7ab592427220a4fc5
I20260925 02:57:09.619714 24426 ts_tablet_manager.cc:1939] T 8b7df659b1e243d7ab592427220a4fc5 P 3ddd2b9c7b714b65bc5d9b20d8d520f7: Deleting consensus metadata
I20260925 02:57:09.621384 23919 catalog_manager.cc:4841] TS 3ddd2b9c7b714b65bc5d9b20d8d520f7 (127.18.196.195:45357): tablet 8b7df659b1e243d7ab592427220a4fc5 (table test-table [id=67e214fee64949328eed2a265ef00f7d]) successfully deleted
I20260925 02:57:09.696630 24427 tablet_replica.cc:330] T 8b7df659b1e243d7ab592427220a4fc5 P d382fd8ba89b4c0e876467508a5b7e84: stopping tablet replica
I20260925 02:57:09.696960 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 23992
W20260925 02:57:09.715293 23918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.196.193:42751 (error 108)
I20260925 02:57:09.715412 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 24125
I20260925 02:57:09.717505 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.716939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:42751 (local address 127.0.0.1:55736)
0925 02:57:09.717186 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:09.717317 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260925 02:57:09.717769 23918 catalog_manager.cc:4584] TS d382fd8ba89b4c0e876467508a5b7e84 (127.18.196.193:42751): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b7df659b1e243d7ab592427220a4fc5: Network error: Client connection negotiation failed: client connection to 127.18.196.193:42751: connect: Connection refused (error 111)
I20260925 02:57:09.718804 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.716460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47968 (local address 127.18.196.194:37925)
0925 02:57:09.716673 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:09.716678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:09.716802 (+ 124us) server_negotiation.cc:407] Connection header received
0925 02:57:09.716990 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:09.716996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:09.717111 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:09.717226 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:09.718191 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:09.718196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:09.718200 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:09.718215 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:09.718249 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:09.718261 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:09.718364 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:09.718477 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 02:57:09.718523 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:57:09.719585 24247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:09.771376 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.770813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:42751 (local address 127.0.0.1:55744)
0925 02:57:09.771085 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:09.771195 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 02:57:09.772555 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.772143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:42751 (local address 127.0.0.1:55748)
0925 02:57:09.772307 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:09.772402 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 02:57:09.807600 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 24125
I20260925 02:57:09.825273 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 24255
I20260925 02:57:09.828413 24399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.826036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58972 (local address 127.18.196.195:45357)
0925 02:57:09.826210 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:09.826215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:09.826429 (+ 214us) server_negotiation.cc:407] Connection header received
0925 02:57:09.826659 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:09.826670 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:09.826727 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:09.826843 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:09.827743 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:09.827750 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:09.827754 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:09.827774 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:09.827828 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:09.827839 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:09.827987 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:09.828068 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 02:57:09.828132 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:57:09.829167 24379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:09.833180 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.832793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:42751 (local address 127.0.0.1:55750)
0925 02:57:09.832978 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:09.833064 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 02:57:09.834370 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.833975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:42751 (local address 127.0.0.1:55754)
0925 02:57:09.834109 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:09.834206 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 02:57:09.914912 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.914411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:42751 (local address 127.0.0.1:55762)
0925 02:57:09.914620 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:09.914710 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 02:57:09.915951 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.915546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:42751 (local address 127.0.0.1:55764)
0925 02:57:09.915733 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:09.915825 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 02:57:09.943851 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 24255
I20260925 02:57:09.963521 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 23902
I20260925 02:57:09.964659 23963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:10.068391 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 23902
2026-09-25T02:57:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2607 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 02:57:10.111486 19219 mini_oidc.cc:152] Starting JWKS server
I20260925 02:57:10.115427 19219 webserver.cc:460] Webserver started at https://127.0.0.1:34569/ using document root <none> and password file <none>
I20260925 02:57:10.115728 19219 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 02:57:10.116335 19219 webserver.cc:460] Webserver started at http://127.0.0.1:35231/ using document root <none> and password file <none>
2026-09-25T02:57:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:57:10Z Disabled control of system clock
I20260925 02:57:10.142485 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:38983
--webserver_interface=127.18.196.254
--webserver_port=0
--builtin_ntp_servers=127.18.196.212:34567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:38983
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34569/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304987644417-19219-0/testchainca.cert with env {}
W20260925 02:57:10.454547 24440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:10.454802 24440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:10.454973 24440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:10.461881 24440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:10.461973 24440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:10.462014 24440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:10.462069 24440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:10.462114 24440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:57:10.462169 24440 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:57:10.462234 24440 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34569/jwks.json
I20260925 02:57:10.471802 24440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:34567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:38983
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:38983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34569/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304987644417-19219-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:10.472925 24440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:10.474088 24440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:10.485407 24446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:10.485455 24445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:10.485473 24448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:10.486686 24440 server_base.cc:1034] running on GCE node
I20260925 02:57:10.487464 24440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:10.488852 24440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:10.490058 24440 hybrid_clock.cc:648] HybridClock initialized: now 1790305030490017 us; error 32 us; skew 500 ppm
I20260925 02:57:10.490394 24440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:57:10.492559 24440 webserver.cc:460] Webserver started at http://127.18.196.254:36081/ using document root <none> and password file <none>
I20260925 02:57:10.493194 24440 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:10.493311 24440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:10.493625 24440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:10.496390 24440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "84b9801fd3224b888c0be27ceff427aa"
format_stamp: "Formatted at 2026-09-25 02:57:10 on dist-test-slave-8zvk"
server_key: "838261380f9b0ff314936031b2daea00"
server_key_iv: "6f1f079d562bf97efeda46dba1ba83f4"
server_key_version: "encryptionkey@0"
I20260925 02:57:10.497135 24440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "84b9801fd3224b888c0be27ceff427aa"
format_stamp: "Formatted at 2026-09-25 02:57:10 on dist-test-slave-8zvk"
server_key: "838261380f9b0ff314936031b2daea00"
server_key_iv: "6f1f079d562bf97efeda46dba1ba83f4"
server_key_version: "encryptionkey@0"
I20260925 02:57:10.501775 24440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 02:57:10.504961 24454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:10.506211 24440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 02:57:10.506381 24440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "84b9801fd3224b888c0be27ceff427aa"
format_stamp: "Formatted at 2026-09-25 02:57:10 on dist-test-slave-8zvk"
server_key: "838261380f9b0ff314936031b2daea00"
server_key_iv: "6f1f079d562bf97efeda46dba1ba83f4"
server_key_version: "encryptionkey@0"
I20260925 02:57:10.506584 24440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:10.517743 24440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:10.518445 24440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:10.518730 24440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:10.536082 24440 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:38983
I20260925 02:57:10.536110 24505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:38983 every 8 connection(s)
I20260925 02:57:10.537282 24440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:57:10.538416 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 24440
I20260925 02:57:10.538623 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:57:10.539428 19219 external_mini_cluster.cc:1428] Setting key a9a84b1225b125d93eb94a1b98f0c02a
I20260925 02:57:10.541563 24506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:10.552556 24509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:10.541155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47620 (local address 127.18.196.254:38983)
0925 02:57:10.542124 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:10.542134 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:10.542171 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:57:10.543809 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:10.543830 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:10.544248 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:10.544562 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:10.545153 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:10.546408 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:10.548321 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:10.548925 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:10.549068 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:10.550281 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:10.550288 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:10.550299 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:10.550827 (+ 528us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:10.550971 (+ 144us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:10.550992 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:10.551083 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:10.551300 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 02:57:10.551632 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":394,"threads_started":1}
I20260925 02:57:10.553862 24506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84b9801fd3224b888c0be27ceff427aa: Bootstrap starting.
I20260925 02:57:10.556130 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:57:10.557024 24506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84b9801fd3224b888c0be27ceff427aa: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:10.557936 24506 log.cc:824] T 00000000000000000000000000000000 P 84b9801fd3224b888c0be27ceff427aa: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:10.560528 24506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84b9801fd3224b888c0be27ceff427aa: No bootstrap required, opened a new log
I20260925 02:57:10.567484 24506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84b9801fd3224b888c0be27ceff427aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84b9801fd3224b888c0be27ceff427aa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 38983 } }
I20260925 02:57:10.567804 24506 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84b9801fd3224b888c0be27ceff427aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:10.567903 24506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84b9801fd3224b888c0be27ceff427aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84b9801fd3224b888c0be27ceff427aa, State: Initialized, Role: FOLLOWER
I20260925 02:57:10.568442 24506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84b9801fd3224b888c0be27ceff427aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84b9801fd3224b888c0be27ceff427aa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 38983 } }
I20260925 02:57:10.568673 24506 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84b9801fd3224b888c0be27ceff427aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:57:10.568809 24506 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84b9801fd3224b888c0be27ceff427aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:57:10.568974 24506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84b9801fd3224b888c0be27ceff427aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:10.570982 24506 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84b9801fd3224b888c0be27ceff427aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84b9801fd3224b888c0be27ceff427aa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 38983 } }
I20260925 02:57:10.571499 24506 leader_election.cc:302] T 00000000000000000000000000000000 P 84b9801fd3224b888c0be27ceff427aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84b9801fd3224b888c0be27ceff427aa; no voters:
I20260925 02:57:10.572104 24506 leader_election.cc:288] T 00000000000000000000000000000000 P 84b9801fd3224b888c0be27ceff427aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:57:10.572233 24511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84b9801fd3224b888c0be27ceff427aa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:10.572901 24511 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84b9801fd3224b888c0be27ceff427aa [term 1 LEADER]: Becoming Leader. State: Replica: 84b9801fd3224b888c0be27ceff427aa, State: Running, Role: LEADER
I20260925 02:57:10.573390 24511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84b9801fd3224b888c0be27ceff427aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84b9801fd3224b888c0be27ceff427aa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 38983 } }
I20260925 02:57:10.573887 24506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84b9801fd3224b888c0be27ceff427aa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:10.576396 24513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84b9801fd3224b888c0be27ceff427aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84b9801fd3224b888c0be27ceff427aa. Latest consensus state: current_term: 1 leader_uuid: "84b9801fd3224b888c0be27ceff427aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84b9801fd3224b888c0be27ceff427aa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 38983 } } }
I20260925 02:57:10.576841 24513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84b9801fd3224b888c0be27ceff427aa [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:10.577596 24512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84b9801fd3224b888c0be27ceff427aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84b9801fd3224b888c0be27ceff427aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84b9801fd3224b888c0be27ceff427aa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 38983 } } }
I20260925 02:57:10.577876 24512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84b9801fd3224b888c0be27ceff427aa [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:10.578434 24517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:10.584095 24517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:10.591877 24517 catalog_manager.cc:1347] Generated new cluster ID: 92d8d5d886124a538ed34d978e8fc051
I20260925 02:57:10.591972 24517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:10.600973 24517 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:57:10.602556 24517 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:10.614825 24517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84b9801fd3224b888c0be27ceff427aa: Generated new TSK 0
I20260925 02:57:10.615504 24517 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 02:57:10.657418 24536 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 02:57:10.657581 24536 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 02:57:10.659591 24509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:10.638320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47624 (local address 127.18.196.254:38983)
0925 02:57:10.638497 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:10.638504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:10.639007 (+ 503us) server_negotiation.cc:407] Connection header received
0925 02:57:10.639254 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:10.639260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:10.639341 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:10.639478 (+ 137us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:57:10.639922 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:10.640933 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:10.642307 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:10.643007 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:10.643148 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:10.643625 (+ 477us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:57:10.658952 (+ 15327us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34569/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 02:57:10.659279 (+ 327us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47624: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34569/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 02:57:10.659709 24509 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47624: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34569/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 02:57:10.660811 24535 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:10.638260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:38983 (local address 127.0.0.1:47624)
0925 02:57:10.638911 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:10.638933 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:10.639059 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:10.639506 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:10.639509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:10.639518 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:57:10.639795 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:10.639802 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:10.641058 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:10.641062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:10.642170 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:10.642181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:10.643179 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:10.643184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:10.643292 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:10.643342 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:57:10.659183 (+ 15841us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34569/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 02:57:10.659439 (+ 256us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.196.254:38983: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34569/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":324,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.18.196.254:38983: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34569/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 02:57:10.662617 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 24440
I20260925 02:57:10.663671 24501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:10.753289 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 24440
2026-09-25T02:57:10Z chronyd exiting
I20260925 02:57:10.803283 19219 test_util.cc:175] -----------------------------------------------
I20260925 02:57:10.803411 19219 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304987644417-19219-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (697 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 02:57:10.807116 19219 mini_oidc.cc:152] Starting JWKS server
I20260925 02:57:10.808907 19219 webserver.cc:460] Webserver started at https://127.0.0.1:38165/ using document root <none> and password file <none>
I20260925 02:57:10.809144 19219 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 02:57:10.809816 19219 webserver.cc:460] Webserver started at http://127.0.0.1:43095/ using document root <none> and password file <none>
2026-09-25T02:57:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:57:10Z Disabled control of system clock
I20260925 02:57:10.835650 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:41127
--webserver_interface=127.18.196.254
--webserver_port=0
--builtin_ntp_servers=127.18.196.212:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:41127
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38165/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304987644417-19219-0/testchainca.cert with env {}
W20260925 02:57:11.140313 24543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:11.140564 24543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:11.140646 24543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:11.147429 24543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:11.147518 24543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:11.147557 24543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:11.147612 24543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:11.147655 24543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:57:11.147725 24543 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:57:11.147763 24543 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38165/jwks.json
I20260925 02:57:11.156946 24543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:41127
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:41127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38165/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304987644417-19219-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:11.157974 24543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:11.159220 24543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:11.170514 24549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:11.170562 24548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:11.170593 24551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:11.171196 24543 server_base.cc:1034] running on GCE node
I20260925 02:57:11.171895 24543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:11.173101 24543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:11.174320 24543 hybrid_clock.cc:648] HybridClock initialized: now 1790305031174278 us; error 54 us; skew 500 ppm
I20260925 02:57:11.174667 24543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:57:11.176921 24543 webserver.cc:460] Webserver started at http://127.18.196.254:37773/ using document root <none> and password file <none>
I20260925 02:57:11.177538 24543 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:11.177667 24543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:11.177965 24543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:11.180615 24543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "c658561198354f53a2aca8bf0392511d"
format_stamp: "Formatted at 2026-09-25 02:57:11 on dist-test-slave-8zvk"
server_key: "4e06ac56f39880839a16a51de96dea0b"
server_key_iv: "5e6aa705d32a5583380f146a508abd22"
server_key_version: "encryptionkey@0"
I20260925 02:57:11.181386 24543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "c658561198354f53a2aca8bf0392511d"
format_stamp: "Formatted at 2026-09-25 02:57:11 on dist-test-slave-8zvk"
server_key: "4e06ac56f39880839a16a51de96dea0b"
server_key_iv: "5e6aa705d32a5583380f146a508abd22"
server_key_version: "encryptionkey@0"
I20260925 02:57:11.186039 24543 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 02:57:11.189349 24557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:11.190541 24543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 02:57:11.190716 24543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "c658561198354f53a2aca8bf0392511d"
format_stamp: "Formatted at 2026-09-25 02:57:11 on dist-test-slave-8zvk"
server_key: "4e06ac56f39880839a16a51de96dea0b"
server_key_iv: "5e6aa705d32a5583380f146a508abd22"
server_key_version: "encryptionkey@0"
I20260925 02:57:11.190941 24543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:11.210309 24543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:11.211077 24543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:11.211371 24543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:11.232226 24543 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:41127
I20260925 02:57:11.232249 24608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:41127 every 8 connection(s)
I20260925 02:57:11.233546 24543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:57:11.237311 24609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:11.240674 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 24543
I20260925 02:57:11.240862 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:57:11.241729 19219 external_mini_cluster.cc:1428] Setting key 642c867cd9b2aaa9b03c8f37c347c021
I20260925 02:57:11.249607 24609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c658561198354f53a2aca8bf0392511d: Bootstrap starting.
I20260925 02:57:11.252584 24609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c658561198354f53a2aca8bf0392511d: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:11.253684 24609 log.cc:824] T 00000000000000000000000000000000 P c658561198354f53a2aca8bf0392511d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:11.255352 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.243660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39924 (local address 127.18.196.254:41127)
0925 02:57:11.244585 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.244600 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.244638 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:57:11.246422 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.246447 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.246967 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.247348 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.247917 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.249135 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.250428 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.251181 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.251379 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:11.252530 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:11.252537 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:11.252550 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:11.253245 (+ 695us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:11.253334 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:11.253356 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:11.253493 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:11.253724 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 02:57:11.254063 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":330,"threads_started":1}
I20260925 02:57:11.256860 24609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c658561198354f53a2aca8bf0392511d: No bootstrap required, opened a new log
I20260925 02:57:11.259337 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:57:11.264951 24609 raft_consensus.cc:348] T 00000000000000000000000000000000 P c658561198354f53a2aca8bf0392511d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c658561198354f53a2aca8bf0392511d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 41127 } }
I20260925 02:57:11.265322 24609 raft_consensus.cc:374] T 00000000000000000000000000000000 P c658561198354f53a2aca8bf0392511d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:11.265439 24609 raft_consensus.cc:729] T 00000000000000000000000000000000 P c658561198354f53a2aca8bf0392511d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c658561198354f53a2aca8bf0392511d, State: Initialized, Role: FOLLOWER
I20260925 02:57:11.265969 24609 consensus_queue.cc:260] T 00000000000000000000000000000000 P c658561198354f53a2aca8bf0392511d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c658561198354f53a2aca8bf0392511d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 41127 } }
I20260925 02:57:11.266213 24609 raft_consensus.cc:388] T 00000000000000000000000000000000 P c658561198354f53a2aca8bf0392511d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:57:11.266361 24609 raft_consensus.cc:482] T 00000000000000000000000000000000 P c658561198354f53a2aca8bf0392511d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:57:11.266532 24609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c658561198354f53a2aca8bf0392511d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:11.269052 24609 raft_consensus.cc:504] T 00000000000000000000000000000000 P c658561198354f53a2aca8bf0392511d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c658561198354f53a2aca8bf0392511d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 41127 } }
I20260925 02:57:11.269558 24609 leader_election.cc:302] T 00000000000000000000000000000000 P c658561198354f53a2aca8bf0392511d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c658561198354f53a2aca8bf0392511d; no voters:
I20260925 02:57:11.270160 24609 leader_election.cc:288] T 00000000000000000000000000000000 P c658561198354f53a2aca8bf0392511d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:57:11.270296 24614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c658561198354f53a2aca8bf0392511d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:11.271031 24614 raft_consensus.cc:686] T 00000000000000000000000000000000 P c658561198354f53a2aca8bf0392511d [term 1 LEADER]: Becoming Leader. State: Replica: c658561198354f53a2aca8bf0392511d, State: Running, Role: LEADER
I20260925 02:57:11.271622 24614 consensus_queue.cc:237] T 00000000000000000000000000000000 P c658561198354f53a2aca8bf0392511d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c658561198354f53a2aca8bf0392511d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 41127 } }
I20260925 02:57:11.272243 24609 sys_catalog.cc:564] T 00000000000000000000000000000000 P c658561198354f53a2aca8bf0392511d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:11.274194 24615 sys_catalog.cc:455] T 00000000000000000000000000000000 P c658561198354f53a2aca8bf0392511d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c658561198354f53a2aca8bf0392511d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c658561198354f53a2aca8bf0392511d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 41127 } } }
I20260925 02:57:11.274639 24615 sys_catalog.cc:458] T 00000000000000000000000000000000 P c658561198354f53a2aca8bf0392511d [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:11.276870 24620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:11.276749 24616 sys_catalog.cc:455] T 00000000000000000000000000000000 P c658561198354f53a2aca8bf0392511d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c658561198354f53a2aca8bf0392511d. Latest consensus state: current_term: 1 leader_uuid: "c658561198354f53a2aca8bf0392511d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c658561198354f53a2aca8bf0392511d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 41127 } } }
I20260925 02:57:11.277055 24616 sys_catalog.cc:458] T 00000000000000000000000000000000 P c658561198354f53a2aca8bf0392511d [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:11.281834 24620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:11.292665 24620 catalog_manager.cc:1347] Generated new cluster ID: 94309fa9f6e54c75af5d2528175122cc
I20260925 02:57:11.292788 24620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:11.306241 24620 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:57:11.308122 24620 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:11.316622 24620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c658561198354f53a2aca8bf0392511d: Generated new TSK 0
I20260925 02:57:11.317456 24620 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 02:57:11.346478 24639 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 02:57:11.346652 24639 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 02:57:11.347851 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.328828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39932 (local address 127.18.196.254:41127)
0925 02:57:11.328993 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.328997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.329474 (+ 477us) server_negotiation.cc:407] Connection header received
0925 02:57:11.329722 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.329726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.329789 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.329935 (+ 146us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:57:11.330402 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.331142 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.332546 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.333009 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.333144 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:11.333630 (+ 486us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:57:11.347149 (+ 13519us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 02:57:11.347552 (+ 403us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39932: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 02:57:11.347994 24638 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:11.328748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:41127 (local address 127.0.0.1:39932)
0925 02:57:11.329356 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:11.329370 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:11.329536 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:11.329960 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:11.329965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:11.329978 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:57:11.330229 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:11.330238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.331284 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.331293 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:11.332391 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:11.332402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.333170 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.333176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:11.333353 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:11.333385 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:57:11.347361 (+ 13976us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 02:57:11.347600 (+ 239us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.196.254:41127: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":380,"threads_started":1}
W20260925 02:57:11.347985 24612 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39932: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260925 02:57:11.349068 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 24543
I20260925 02:57:11.350263 24604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:11.440222 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 24543
2026-09-25T02:57:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (687 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 02:57:11.494643 19219 mini_oidc.cc:152] Starting JWKS server
I20260925 02:57:11.496443 19219 webserver.cc:460] Webserver started at https://127.0.0.1:44469/ using document root <none> and password file <none>
I20260925 02:57:11.496632 19219 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 02:57:11.497164 19219 webserver.cc:460] Webserver started at http://127.0.0.1:44711/ using document root <none> and password file <none>
2026-09-25T02:57:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:57:11Z Disabled control of system clock
I20260925 02:57:11.522531 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:40139
--webserver_interface=127.18.196.254
--webserver_port=0
--builtin_ntp_servers=127.18.196.212:37625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:40139
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44469/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 02:57:11.833258 24646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:11.833673 24646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:11.833850 24646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:11.843999 24646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:11.844128 24646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:11.844197 24646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:11.844288 24646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:11.844355 24646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:57:11.844455 24646 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:57:11.844550 24646 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44469/jwks.json
I20260925 02:57:11.858110 24646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:37625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:40139
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:40139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44469/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:11.859710 24646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:11.861400 24646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:11.872159 24651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:11.872393 24652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:11.872807 24654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:11.874249 24646 server_base.cc:1034] running on GCE node
I20260925 02:57:11.875245 24646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:11.877043 24646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:11.878269 24646 hybrid_clock.cc:648] HybridClock initialized: now 1790305031878224 us; error 41 us; skew 500 ppm
I20260925 02:57:11.878546 24646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:57:11.880666 24646 webserver.cc:460] Webserver started at http://127.18.196.254:42773/ using document root <none> and password file <none>
I20260925 02:57:11.881207 24646 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:11.881279 24646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:11.881565 24646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:11.884232 24646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "a22563d614ea4bcc85537f255e1bbe43"
format_stamp: "Formatted at 2026-09-25 02:57:11 on dist-test-slave-8zvk"
server_key: "3e51e748e36d772bdfff435e607160e0"
server_key_iv: "57e90aeb9ff95c688fe476b08aaaead7"
server_key_version: "encryptionkey@0"
I20260925 02:57:11.884966 24646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "a22563d614ea4bcc85537f255e1bbe43"
format_stamp: "Formatted at 2026-09-25 02:57:11 on dist-test-slave-8zvk"
server_key: "3e51e748e36d772bdfff435e607160e0"
server_key_iv: "57e90aeb9ff95c688fe476b08aaaead7"
server_key_version: "encryptionkey@0"
I20260925 02:57:11.889624 24646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 02:57:11.892750 24660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:11.893941 24646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:57:11.894071 24646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "a22563d614ea4bcc85537f255e1bbe43"
format_stamp: "Formatted at 2026-09-25 02:57:11 on dist-test-slave-8zvk"
server_key: "3e51e748e36d772bdfff435e607160e0"
server_key_iv: "57e90aeb9ff95c688fe476b08aaaead7"
server_key_version: "encryptionkey@0"
I20260925 02:57:11.894238 24646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:11.923696 24646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:11.924419 24646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:11.924664 24646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:11.941642 24646 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:40139
I20260925 02:57:11.941650 24711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:40139 every 8 connection(s)
I20260925 02:57:11.942904 24646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:57:11.946530 24712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:11.948381 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 24646
I20260925 02:57:11.948597 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:57:11.949376 19219 external_mini_cluster.cc:1428] Setting key 147bcd62c9475d01f5d569744a5b4aca
I20260925 02:57:11.960845 24712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a22563d614ea4bcc85537f255e1bbe43: Bootstrap starting.
I20260925 02:57:11.962352 24715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.951076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39236 (local address 127.18.196.254:40139)
0925 02:57:11.952129 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.952142 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.952187 (+ 45us) server_negotiation.cc:407] Connection header received
0925 02:57:11.953653 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.953674 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.954114 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.954588 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.955120 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.956352 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.958214 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.958925 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.959073 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:11.960221 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:11.960229 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:11.960241 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:11.960726 (+ 485us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:11.960794 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:11.960818 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:11.960905 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:11.961161 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 02:57:11.961489 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":383,"threads_started":1}
I20260925 02:57:11.964071 24712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a22563d614ea4bcc85537f255e1bbe43: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:11.965209 24712 log.cc:824] T 00000000000000000000000000000000 P a22563d614ea4bcc85537f255e1bbe43: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:11.966238 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:57:11.968030 24712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a22563d614ea4bcc85537f255e1bbe43: No bootstrap required, opened a new log
I20260925 02:57:11.976007 24712 raft_consensus.cc:348] T 00000000000000000000000000000000 P a22563d614ea4bcc85537f255e1bbe43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22563d614ea4bcc85537f255e1bbe43" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40139 } }
I20260925 02:57:11.976336 24712 raft_consensus.cc:374] T 00000000000000000000000000000000 P a22563d614ea4bcc85537f255e1bbe43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:11.976454 24712 raft_consensus.cc:729] T 00000000000000000000000000000000 P a22563d614ea4bcc85537f255e1bbe43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22563d614ea4bcc85537f255e1bbe43, State: Initialized, Role: FOLLOWER
I20260925 02:57:11.976955 24712 consensus_queue.cc:260] T 00000000000000000000000000000000 P a22563d614ea4bcc85537f255e1bbe43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22563d614ea4bcc85537f255e1bbe43" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40139 } }
I20260925 02:57:11.977182 24712 raft_consensus.cc:388] T 00000000000000000000000000000000 P a22563d614ea4bcc85537f255e1bbe43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:57:11.977321 24712 raft_consensus.cc:482] T 00000000000000000000000000000000 P a22563d614ea4bcc85537f255e1bbe43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:57:11.977499 24712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a22563d614ea4bcc85537f255e1bbe43 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:11.979563 24712 raft_consensus.cc:504] T 00000000000000000000000000000000 P a22563d614ea4bcc85537f255e1bbe43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22563d614ea4bcc85537f255e1bbe43" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40139 } }
I20260925 02:57:11.980057 24712 leader_election.cc:302] T 00000000000000000000000000000000 P a22563d614ea4bcc85537f255e1bbe43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a22563d614ea4bcc85537f255e1bbe43; no voters:
I20260925 02:57:11.980602 24712 leader_election.cc:288] T 00000000000000000000000000000000 P a22563d614ea4bcc85537f255e1bbe43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:57:11.980731 24717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a22563d614ea4bcc85537f255e1bbe43 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:11.981402 24717 raft_consensus.cc:686] T 00000000000000000000000000000000 P a22563d614ea4bcc85537f255e1bbe43 [term 1 LEADER]: Becoming Leader. State: Replica: a22563d614ea4bcc85537f255e1bbe43, State: Running, Role: LEADER
I20260925 02:57:11.981921 24717 consensus_queue.cc:237] T 00000000000000000000000000000000 P a22563d614ea4bcc85537f255e1bbe43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22563d614ea4bcc85537f255e1bbe43" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40139 } }
I20260925 02:57:11.982764 24712 sys_catalog.cc:564] T 00000000000000000000000000000000 P a22563d614ea4bcc85537f255e1bbe43 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:11.985175 24719 sys_catalog.cc:455] T 00000000000000000000000000000000 P a22563d614ea4bcc85537f255e1bbe43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a22563d614ea4bcc85537f255e1bbe43. Latest consensus state: current_term: 1 leader_uuid: "a22563d614ea4bcc85537f255e1bbe43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22563d614ea4bcc85537f255e1bbe43" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40139 } } }
I20260925 02:57:11.985888 24719 sys_catalog.cc:458] T 00000000000000000000000000000000 P a22563d614ea4bcc85537f255e1bbe43 [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:11.985188 24718 sys_catalog.cc:455] T 00000000000000000000000000000000 P a22563d614ea4bcc85537f255e1bbe43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a22563d614ea4bcc85537f255e1bbe43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22563d614ea4bcc85537f255e1bbe43" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40139 } } }
I20260925 02:57:11.986326 24718 sys_catalog.cc:458] T 00000000000000000000000000000000 P a22563d614ea4bcc85537f255e1bbe43 [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:11.996022 24726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:12.000051 24726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:12.007321 24726 catalog_manager.cc:1347] Generated new cluster ID: a972209fab5b49a8adf13a45c36940f0
I20260925 02:57:12.007427 24726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:12.040421 24726 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:57:12.042253 24726 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:12.049136 24726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a22563d614ea4bcc85537f255e1bbe43: Generated new TSK 0
I20260925 02:57:12.050093 24726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:57:12.076102 24715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:12.067466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39242 (local address 127.18.196.254:40139)
0925 02:57:12.067633 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:12.067638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:12.068079 (+ 441us) server_negotiation.cc:407] Connection header received
0925 02:57:12.068276 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:12.068282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:12.068380 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:12.068530 (+ 150us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:57:12.069223 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:12.070348 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:12.071750 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:12.072491 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:12.072619 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:12.073142 (+ 523us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:57:12.075476 (+ 2334us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44469/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 02:57:12.075801 (+ 325us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39242: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44469/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 02:57:12.076256 24715 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39242: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44469/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 02:57:12.077826 24741 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:12.067323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40139 (local address 127.0.0.1:39242)
0925 02:57:12.067982 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:12.068000 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:12.068108 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:12.068577 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:12.068581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:12.068591 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:57:12.069062 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:12.069077 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:12.070488 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:12.070492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:12.071615 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:12.071626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:12.072714 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:12.072720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:12.072847 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:12.072879 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:57:12.075676 (+ 2797us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44469/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 02:57:12.077413 (+ 1737us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.196.254:40139: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44469/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":1446,"thread_start_us":373,"threads_started":1}
I20260925 02:57:12.079207 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 24646
I20260925 02:57:12.080404 24707 generic_service.cc:196] Checking for leaks (request via RPC)
E20260925 02:57:12.087936 24742 webserver.cc:533] Webserver: sslize failed:
E20260925 02:57:12.088022 24742 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 02:57:12.170584 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 24646
2026-09-25T02:57:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (731 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T02:57:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:57:12Z Disabled control of system clock
I20260925 02:57:12.252152 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:33053
--webserver_interface=127.18.196.254
--webserver_port=0
--builtin_ntp_servers=127.18.196.212:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:33053
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304987644417-19219-0/testchainca.cert with env {}
W20260925 02:57:12.559716 24747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:12.559983 24747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:12.560065 24747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:12.566828 24747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:12.566929 24747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:12.566967 24747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:12.567013 24747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:12.567049 24747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:57:12.567134 24747 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:57:12.567196 24747 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 02:57:12.576267 24747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:33053
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:33053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304987644417-19219-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:12.577482 24747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:12.578657 24747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:12.589921 24752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:12.589967 24753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:12.589967 24755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:12.591012 24747 server_base.cc:1034] running on GCE node
I20260925 02:57:12.591727 24747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:12.593026 24747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:12.594249 24747 hybrid_clock.cc:648] HybridClock initialized: now 1790305032594211 us; error 50 us; skew 500 ppm
I20260925 02:57:12.594594 24747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:57:12.596730 24747 webserver.cc:460] Webserver started at http://127.18.196.254:44167/ using document root <none> and password file <none>
I20260925 02:57:12.597361 24747 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:12.597498 24747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:12.597777 24747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:12.600528 24747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "be50b30e7fa04f6f9c3b54f81fceec21"
format_stamp: "Formatted at 2026-09-25 02:57:12 on dist-test-slave-8zvk"
server_key: "b4921bfc9df0f1d514274756351a1ab0"
server_key_iv: "8eb93c2d5224a78dc8471b12b264e038"
server_key_version: "encryptionkey@0"
I20260925 02:57:12.601313 24747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "be50b30e7fa04f6f9c3b54f81fceec21"
format_stamp: "Formatted at 2026-09-25 02:57:12 on dist-test-slave-8zvk"
server_key: "b4921bfc9df0f1d514274756351a1ab0"
server_key_iv: "8eb93c2d5224a78dc8471b12b264e038"
server_key_version: "encryptionkey@0"
I20260925 02:57:12.606045 24747 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 02:57:12.609197 24761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:12.610569 24747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:57:12.610780 24747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "be50b30e7fa04f6f9c3b54f81fceec21"
format_stamp: "Formatted at 2026-09-25 02:57:12 on dist-test-slave-8zvk"
server_key: "b4921bfc9df0f1d514274756351a1ab0"
server_key_iv: "8eb93c2d5224a78dc8471b12b264e038"
server_key_version: "encryptionkey@0"
I20260925 02:57:12.611078 24747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:12.624315 24747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:12.625012 24747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:12.625274 24747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:12.642820 24747 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:33053
I20260925 02:57:12.642882 24812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:33053 every 8 connection(s)
I20260925 02:57:12.644106 24747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:57:12.647552 24813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:12.648628 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 24747
I20260925 02:57:12.648821 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:57:12.649456 19219 external_mini_cluster.cc:1428] Setting key 9eb831d6b7dadbff3e0d6d7c1f30309a
I20260925 02:57:12.660785 24813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be50b30e7fa04f6f9c3b54f81fceec21: Bootstrap starting.
I20260925 02:57:12.662556 24816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:12.651217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48530 (local address 127.18.196.254:33053)
0925 02:57:12.652186 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:12.652196 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:12.652242 (+ 46us) server_negotiation.cc:407] Connection header received
0925 02:57:12.653744 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:12.653764 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:12.654195 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:12.654495 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:12.655091 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:12.656443 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:12.658286 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:12.659016 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:12.659139 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:12.660398 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:12.660406 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:12.660418 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:12.660935 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:12.661020 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:12.661041 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:12.661127 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:12.661379 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 02:57:12.661706 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":362,"threads_started":1}
I20260925 02:57:12.664309 24813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be50b30e7fa04f6f9c3b54f81fceec21: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:12.665472 24813 log.cc:824] T 00000000000000000000000000000000 P be50b30e7fa04f6f9c3b54f81fceec21: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:12.666326 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:57:12.668603 24813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be50b30e7fa04f6f9c3b54f81fceec21: No bootstrap required, opened a new log
I20260925 02:57:12.676015 24813 raft_consensus.cc:348] T 00000000000000000000000000000000 P be50b30e7fa04f6f9c3b54f81fceec21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be50b30e7fa04f6f9c3b54f81fceec21" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33053 } }
I20260925 02:57:12.676316 24813 raft_consensus.cc:374] T 00000000000000000000000000000000 P be50b30e7fa04f6f9c3b54f81fceec21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:12.676388 24813 raft_consensus.cc:729] T 00000000000000000000000000000000 P be50b30e7fa04f6f9c3b54f81fceec21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be50b30e7fa04f6f9c3b54f81fceec21, State: Initialized, Role: FOLLOWER
I20260925 02:57:12.676898 24813 consensus_queue.cc:260] T 00000000000000000000000000000000 P be50b30e7fa04f6f9c3b54f81fceec21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be50b30e7fa04f6f9c3b54f81fceec21" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33053 } }
I20260925 02:57:12.677091 24813 raft_consensus.cc:388] T 00000000000000000000000000000000 P be50b30e7fa04f6f9c3b54f81fceec21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:57:12.677198 24813 raft_consensus.cc:482] T 00000000000000000000000000000000 P be50b30e7fa04f6f9c3b54f81fceec21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:57:12.677310 24813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be50b30e7fa04f6f9c3b54f81fceec21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:12.679625 24813 raft_consensus.cc:504] T 00000000000000000000000000000000 P be50b30e7fa04f6f9c3b54f81fceec21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be50b30e7fa04f6f9c3b54f81fceec21" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33053 } }
I20260925 02:57:12.680032 24813 leader_election.cc:302] T 00000000000000000000000000000000 P be50b30e7fa04f6f9c3b54f81fceec21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be50b30e7fa04f6f9c3b54f81fceec21; no voters:
I20260925 02:57:12.680523 24813 leader_election.cc:288] T 00000000000000000000000000000000 P be50b30e7fa04f6f9c3b54f81fceec21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:57:12.680680 24818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be50b30e7fa04f6f9c3b54f81fceec21 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:12.681454 24818 raft_consensus.cc:686] T 00000000000000000000000000000000 P be50b30e7fa04f6f9c3b54f81fceec21 [term 1 LEADER]: Becoming Leader. State: Replica: be50b30e7fa04f6f9c3b54f81fceec21, State: Running, Role: LEADER
I20260925 02:57:12.682027 24818 consensus_queue.cc:237] T 00000000000000000000000000000000 P be50b30e7fa04f6f9c3b54f81fceec21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be50b30e7fa04f6f9c3b54f81fceec21" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33053 } }
I20260925 02:57:12.682548 24813 sys_catalog.cc:564] T 00000000000000000000000000000000 P be50b30e7fa04f6f9c3b54f81fceec21 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:12.684916 24820 sys_catalog.cc:455] T 00000000000000000000000000000000 P be50b30e7fa04f6f9c3b54f81fceec21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be50b30e7fa04f6f9c3b54f81fceec21. Latest consensus state: current_term: 1 leader_uuid: "be50b30e7fa04f6f9c3b54f81fceec21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be50b30e7fa04f6f9c3b54f81fceec21" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33053 } } }
I20260925 02:57:12.685374 24820 sys_catalog.cc:458] T 00000000000000000000000000000000 P be50b30e7fa04f6f9c3b54f81fceec21 [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:12.684942 24819 sys_catalog.cc:455] T 00000000000000000000000000000000 P be50b30e7fa04f6f9c3b54f81fceec21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be50b30e7fa04f6f9c3b54f81fceec21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be50b30e7fa04f6f9c3b54f81fceec21" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33053 } } }
I20260925 02:57:12.685478 24819 sys_catalog.cc:458] T 00000000000000000000000000000000 P be50b30e7fa04f6f9c3b54f81fceec21 [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:12.687080 24825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:12.692250 24825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:12.700917 24825 catalog_manager.cc:1347] Generated new cluster ID: ada5022fe5ca4b7ea65a9ff3b9b88e7b
I20260925 02:57:12.701025 24825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:12.721515 24825 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:57:12.722697 24825 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:12.731582 24825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be50b30e7fa04f6f9c3b54f81fceec21: Generated new TSK 0
I20260925 02:57:12.732333 24825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:57:12.796407 24816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:12.749758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48544 (local address 127.18.196.254:33053)
0925 02:57:12.750012 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:12.750021 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:12.750551 (+ 530us) server_negotiation.cc:407] Connection header received
0925 02:57:12.750736 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:12.750742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:12.750832 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:12.751048 (+ 216us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:57:12.751423 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:12.752553 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:12.753895 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:12.754579 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:12.754708 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:12.755220 (+ 512us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:57:12.795655 (+ 40435us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 02:57:12.796066 (+ 411us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48544: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 02:57:12.796545 24816 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48544: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 02:57:12.796535 24842 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:12.749638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:33053 (local address 127.0.0.1:48544)
0925 02:57:12.750455 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:12.750470 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:12.750582 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:12.751047 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:12.751051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:12.751060 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:57:12.751285 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:12.751292 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:12.752695 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:12.752698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:12.753737 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:12.753748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:12.754748 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:12.754753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:12.754937 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:12.754977 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:57:12.795956 (+ 40979us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 02:57:12.796207 (+ 251us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.196.254:33053: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":571,"threads_started":1}
I20260925 02:57:12.797598 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 24747
I20260925 02:57:12.799018 24808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:12.894236 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 24747
2026-09-25T02:57:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (705 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (397 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (394 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:57:13 dist-test-slave-8zvk krb5kdc[24857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:57:13 dist-test-slave-8zvk krb5kdc[24857](info): set up 2 sockets
Sep 25 02:57:13 dist-test-slave-8zvk krb5kdc[24857](info): commencing operation
krb5kdc: starting...
W20260925 02:57:14.770906 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 02:57:14 dist-test-slave-8zvk krb5kdc[24857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:57:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:57:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 02:57:14.971902 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:41505
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:41505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:15.293179 24870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:15.293458 24870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:15.293622 24870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:15.300969 24870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:15.301070 24870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:15.301182 24870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:15.301293 24870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:15.301388 24870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:15.311131 24870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:41505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:41505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:15.312429 24870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:15.313668 24870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:15.325676 24875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:15.325716 24878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:15.325757 24876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:15.326555 24870 server_base.cc:1034] running on GCE node
I20260925 02:57:15.327359 24870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:15.328750 24870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:15.329962 24870 hybrid_clock.cc:648] HybridClock initialized: now 1790305035329912 us; error 34 us; skew 500 ppm
I20260925 02:57:15.330291 24870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:15 dist-test-slave-8zvk krb5kdc[24857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305035, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:15.333405 24870 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 02:57:15.335143 24870 webserver.cc:460] Webserver started at http://127.18.196.254:37425/ using document root <none> and password file <none>
I20260925 02:57:15.335822 24870 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:15.335927 24870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:15.336248 24870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:15.339340 24870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "919163eaec4b4416beccb0a6055d55aa"
format_stamp: "Formatted at 2026-09-25 02:57:15 on dist-test-slave-8zvk"
server_key: "078f0ccfc1f804f9992e60f002ec3aed"
server_key_iv: "c857ac123cf2c73bcc97546ad09b9304"
server_key_version: "encryptionkey@0"
I20260925 02:57:15.340139 24870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "919163eaec4b4416beccb0a6055d55aa"
format_stamp: "Formatted at 2026-09-25 02:57:15 on dist-test-slave-8zvk"
server_key: "078f0ccfc1f804f9992e60f002ec3aed"
server_key_iv: "c857ac123cf2c73bcc97546ad09b9304"
server_key_version: "encryptionkey@0"
I20260925 02:57:15.345280 24870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:57:15.348719 24885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:15.350044 24870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:57:15.350332 24870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "919163eaec4b4416beccb0a6055d55aa"
format_stamp: "Formatted at 2026-09-25 02:57:15 on dist-test-slave-8zvk"
server_key: "078f0ccfc1f804f9992e60f002ec3aed"
server_key_iv: "c857ac123cf2c73bcc97546ad09b9304"
server_key_version: "encryptionkey@0"
I20260925 02:57:15.350548 24870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:15.373879 24870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:15.378067 24870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:15.378360 24870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:15.397387 24870 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:41505
I20260925 02:57:15.397426 24936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:41505 every 8 connection(s)
I20260925 02:57:15.398774 24870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:57:15.401180 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 24870
I20260925 02:57:15.401453 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:57:15.402288 19219 external_mini_cluster.cc:1428] Setting key 2da526e5ebd22ed3b3044ada28c610c7
I20260925 02:57:15.403187 24937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 02:57:15 dist-test-slave-8zvk krb5kdc[24857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:57:15.418316 24937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 919163eaec4b4416beccb0a6055d55aa: Bootstrap starting.
I20260925 02:57:15.421507 24937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 919163eaec4b4416beccb0a6055d55aa: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:15.422530 24937 log.cc:824] T 00000000000000000000000000000000 P 919163eaec4b4416beccb0a6055d55aa: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:15.425493 24937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 919163eaec4b4416beccb0a6055d55aa: No bootstrap required, opened a new log
I20260925 02:57:15.425621 24940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:15.404357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59094 (local address 127.18.196.254:41505)
0925 02:57:15.405114 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:15.405126 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:15.405166 (+ 40us) server_negotiation.cc:407] Connection header received
0925 02:57:15.406691 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:15.406712 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:15.407171 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:15.407504 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:15.409330 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:15.411140 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:15.412920 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:15.413591 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:15.413729 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:15.418431 (+ 4702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:15.418446 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:15.418466 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:15.418514 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:15.421537 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:15.422250 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:15.422257 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:15.422263 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:15.422428 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:15.422867 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:15.422872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:15.422875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:15.423454 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:15.423760 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:15.424244 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 02:57:15.424656 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":341,"threads_started":1}
I20260925 02:57:15.432955 24937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 919163eaec4b4416beccb0a6055d55aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919163eaec4b4416beccb0a6055d55aa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 41505 } }
I20260925 02:57:15.433332 24937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 919163eaec4b4416beccb0a6055d55aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:15.433436 24937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 919163eaec4b4416beccb0a6055d55aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 919163eaec4b4416beccb0a6055d55aa, State: Initialized, Role: FOLLOWER
I20260925 02:57:15.434043 24937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 919163eaec4b4416beccb0a6055d55aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919163eaec4b4416beccb0a6055d55aa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 41505 } }
I20260925 02:57:15.434317 24937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 919163eaec4b4416beccb0a6055d55aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:57:15.434463 24937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 919163eaec4b4416beccb0a6055d55aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:57:15.434656 24937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 919163eaec4b4416beccb0a6055d55aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:15.436791 24937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 919163eaec4b4416beccb0a6055d55aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919163eaec4b4416beccb0a6055d55aa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 41505 } }
I20260925 02:57:15.437300 24937 leader_election.cc:302] T 00000000000000000000000000000000 P 919163eaec4b4416beccb0a6055d55aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 919163eaec4b4416beccb0a6055d55aa; no voters:
I20260925 02:57:15.437951 24937 leader_election.cc:288] T 00000000000000000000000000000000 P 919163eaec4b4416beccb0a6055d55aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:57:15.438093 24942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 919163eaec4b4416beccb0a6055d55aa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:15.438762 24942 raft_consensus.cc:686] T 00000000000000000000000000000000 P 919163eaec4b4416beccb0a6055d55aa [term 1 LEADER]: Becoming Leader. State: Replica: 919163eaec4b4416beccb0a6055d55aa, State: Running, Role: LEADER
I20260925 02:57:15.439253 24942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 919163eaec4b4416beccb0a6055d55aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919163eaec4b4416beccb0a6055d55aa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 41505 } }
I20260925 02:57:15.440025 24937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 919163eaec4b4416beccb0a6055d55aa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:15.442176 24944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 919163eaec4b4416beccb0a6055d55aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 919163eaec4b4416beccb0a6055d55aa. Latest consensus state: current_term: 1 leader_uuid: "919163eaec4b4416beccb0a6055d55aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919163eaec4b4416beccb0a6055d55aa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 41505 } } }
I20260925 02:57:15.442596 24944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 919163eaec4b4416beccb0a6055d55aa [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:15.442205 24943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 919163eaec4b4416beccb0a6055d55aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "919163eaec4b4416beccb0a6055d55aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919163eaec4b4416beccb0a6055d55aa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 41505 } } }
I20260925 02:57:15.443222 24943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 919163eaec4b4416beccb0a6055d55aa [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:15.444325 24949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:15.450208 24949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:15.458644 24949 catalog_manager.cc:1347] Generated new cluster ID: 4b51542957264669b08a3d1d61250a94
I20260925 02:57:15.458750 24949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:15.476413 24949 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:57:15.477770 24949 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:15.485657 24949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 919163eaec4b4416beccb0a6055d55aa: Generated new TSK 0
I20260925 02:57:15.486443 24949 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
I20260925 02:57:15.546360 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:41505
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:15.922792 24963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:15.923349 24963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:15.923741 24963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:15.937049 24963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:15.937317 24963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:15.937630 24963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 02:57:15.957500 24963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:41505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:15.960237 24963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:15.963047 24963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:15.986444 24968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:15.986501 24969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:15.986927 24971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:15.987869 24963 server_base.cc:1034] running on GCE node
I20260925 02:57:15.989133 24963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:15.990585 24963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:15.991912 24963 hybrid_clock.cc:648] HybridClock initialized: now 1790305035991816 us; error 120 us; skew 500 ppm
I20260925 02:57:15.992362 24963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:15 dist-test-slave-8zvk krb5kdc[24857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305035, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:15.997643 24963 init.cc:377] Logged in from keytab as kudu/127.18.196.193@KRBTEST.COM (short username kudu)
I20260925 02:57:16.000303 24963 webserver.cc:460] Webserver started at http://127.18.196.193:36563/ using document root <none> and password file <none>
I20260925 02:57:16.001397 24963 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:16.001569 24963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:16.001961 24963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:16.007486 24963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "d4c99748f2f743e8a30a3a711ca89e42"
format_stamp: "Formatted at 2026-09-25 02:57:15 on dist-test-slave-8zvk"
server_key: "73cc97b6fb7c1d50192aa59042076bbd"
server_key_iv: "1ed9d8a8a104d88fea7afbd9ad2c3934"
server_key_version: "encryptionkey@0"
I20260925 02:57:16.008920 24963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "d4c99748f2f743e8a30a3a711ca89e42"
format_stamp: "Formatted at 2026-09-25 02:57:15 on dist-test-slave-8zvk"
server_key: "73cc97b6fb7c1d50192aa59042076bbd"
server_key_iv: "1ed9d8a8a104d88fea7afbd9ad2c3934"
server_key_version: "encryptionkey@0"
I20260925 02:57:16.018266 24963 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s
I20260925 02:57:16.024706 24978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:16.027079 24963 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260925 02:57:16.027284 24963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "d4c99748f2f743e8a30a3a711ca89e42"
format_stamp: "Formatted at 2026-09-25 02:57:15 on dist-test-slave-8zvk"
server_key: "73cc97b6fb7c1d50192aa59042076bbd"
server_key_iv: "1ed9d8a8a104d88fea7afbd9ad2c3934"
server_key_version: "encryptionkey@0"
I20260925 02:57:16.027647 24963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:16.068579 24963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:16.075397 24963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:16.075858 24963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:16.077277 24963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:16.079442 24963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:16.079607 24963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:16.079752 24963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:16.079843 24963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:16.140013 24963 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:38167
I20260925 02:57:16.140136 25090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:38167 every 8 connection(s)
I20260925 02:57:16.142256 24963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 02:57:16.153339 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 24963
I20260925 02:57:16.153784 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 02:57:16.155505 19219 external_mini_cluster.cc:1428] Setting key 59e6bd9cd156377a33008fba682d4197
Sep 25 02:57:16 dist-test-slave-8zvk krb5kdc[24857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305035, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:57:16.180953 25093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:16.146749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:60151 (local address 127.18.196.254:41505)
0925 02:57:16.147422 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:16.147435 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:16.149086 (+ 1651us) server_negotiation.cc:407] Connection header received
0925 02:57:16.152819 (+ 3733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:16.152832 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:16.152943 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:16.153111 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:16.157240 (+ 4129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:16.158433 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:16.161312 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:16.162029 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:16.162331 (+ 302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:16.168416 (+ 6085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:16.168429 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:16.168433 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:16.168490 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:16.174260 (+ 5770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:16.175752 (+ 1492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:16.175772 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:16.175778 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:16.176071 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:16.177090 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:16.177109 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:16.177115 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:16.177894 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:16.178196 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:16.179547 (+ 1351us) server_negotiation.cc:299] Negotiation successful
0925 02:57:16.179984 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":317,"threads_started":1}
I20260925 02:57:16.181643 25094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:16.147726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:41505 (local address 127.18.196.193:60151)
0925 02:57:16.148838 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:16.148913 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:16.152099 (+ 3186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:16.153710 (+ 1611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:16.153743 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:16.155243 (+ 1500us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:16.156874 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:16.156920 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:16.158764 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:16.158786 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:16.161033 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:16.161061 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:16.162466 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:16.162486 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:16.163076 (+ 590us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:16.164268 (+ 1192us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:16.164306 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:16.168061 (+ 3755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:16.174573 (+ 6512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:16.174581 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:16.174609 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:16.175536 (+ 927us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:16.176290 (+ 754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:16.176321 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:16.176326 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:16.176853 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:16.178095 (+ 1242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:16.178106 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:16.178620 (+ 514us) client_negotiation.cc:769] Sending connection context
0925 02:57:16.179131 (+ 511us) client_negotiation.cc:240] Negotiation successful
0925 02:57:16.179857 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":449,"threads_started":1}
I20260925 02:57:16.184237 25091 heartbeater.cc:347] Connected to a master server at 127.18.196.254:41505
I20260925 02:57:16.184810 25091 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:16.186401 25091 heartbeater.cc:511] Master 127.18.196.254:41505 requested a full tablet report, sending...
I20260925 02:57:16.190621 24902 ts_manager.cc:194] Registered new tserver with Master: d4c99748f2f743e8a30a3a711ca89e42 (127.18.196.193:38167)
WARNING: no policy specified for kudu/127.18.196.194@KRBTEST.COM; defaulting to no policy
I20260925 02:57:16.193058 24902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.193@KRBTEST.COM'} at 127.18.196.193:60151
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
I20260925 02:57:16.229012 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:41505
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:16.641848 25097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:16.642158 25097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:16.642326 25097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:16.649523 25097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:16.649686 25097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:16.649863 25097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 02:57:16.659765 25097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:41505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:16.661062 25097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:16.662375 25097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:16.675527 25102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:16.675607 25105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:16.675565 25103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:16.676494 25097 server_base.cc:1034] running on GCE node
I20260925 02:57:16.677188 25097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:16.678166 25097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:16.679359 25097 hybrid_clock.cc:648] HybridClock initialized: now 1790305036679292 us; error 59 us; skew 500 ppm
I20260925 02:57:16.679674 25097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:16 dist-test-slave-8zvk krb5kdc[24857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305036, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:16.682928 25097 init.cc:377] Logged in from keytab as kudu/127.18.196.194@KRBTEST.COM (short username kudu)
I20260925 02:57:16.684244 25097 webserver.cc:460] Webserver started at http://127.18.196.194:36113/ using document root <none> and password file <none>
I20260925 02:57:16.684866 25097 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:16.684970 25097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:16.685213 25097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:16.688099 25097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "e4701f0ee9c0441fb467fd4084b3e378"
format_stamp: "Formatted at 2026-09-25 02:57:16 on dist-test-slave-8zvk"
server_key: "5c7a43cfda9291b792cacc281eafed8f"
server_key_iv: "59d1f6d85b98954c6ca1c4b8d8641929"
server_key_version: "encryptionkey@0"
I20260925 02:57:16.688822 25097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "e4701f0ee9c0441fb467fd4084b3e378"
format_stamp: "Formatted at 2026-09-25 02:57:16 on dist-test-slave-8zvk"
server_key: "5c7a43cfda9291b792cacc281eafed8f"
server_key_iv: "59d1f6d85b98954c6ca1c4b8d8641929"
server_key_version: "encryptionkey@0"
I20260925 02:57:16.693763 25097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:57:16.697031 25112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:16.698282 25097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:57:16.698442 25097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "e4701f0ee9c0441fb467fd4084b3e378"
format_stamp: "Formatted at 2026-09-25 02:57:16 on dist-test-slave-8zvk"
server_key: "5c7a43cfda9291b792cacc281eafed8f"
server_key_iv: "59d1f6d85b98954c6ca1c4b8d8641929"
server_key_version: "encryptionkey@0"
I20260925 02:57:16.698643 25097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:16.720953 25097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:16.725173 25097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:16.725455 25097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:16.726384 25097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:16.727648 25097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:16.727721 25097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:16.727859 25097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:16.727952 25097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:16.761941 25097 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:34611
I20260925 02:57:16.761991 25224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:34611 every 8 connection(s)
I20260925 02:57:16.763298 25097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 02:57:16.767787 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 25097
I20260925 02:57:16.767966 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 02:57:16.768546 19219 external_mini_cluster.cc:1428] Setting key 765069e5f0b8bb9db8e0e6023485c7a5
Sep 25 02:57:16 dist-test-slave-8zvk krb5kdc[24857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305036, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:57:16.785785 25227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:16.765886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:50965 (local address 127.18.196.254:41505)
0925 02:57:16.766341 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:16.766347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:16.767195 (+ 848us) server_negotiation.cc:407] Connection header received
0925 02:57:16.769605 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:16.769612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:16.769708 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:16.769823 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:16.771692 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:16.772455 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:16.774938 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:16.775400 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:16.775515 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:16.779988 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:16.779993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:16.779996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:16.780028 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:16.782582 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:16.783543 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:16.783547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:16.783549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:16.783671 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:16.784142 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:16.784145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:16.784147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:16.784403 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:16.784519 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:16.785368 (+ 849us) server_negotiation.cc:299] Negotiation successful
0925 02:57:16.785530 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":246,"threads_started":1}
I20260925 02:57:16.787464 25228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:16.766281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:41505 (local address 127.18.196.194:50965)
0925 02:57:16.767035 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:16.767072 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:16.769345 (+ 2273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:16.770049 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:16.770057 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:16.770613 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:16.771434 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:16.771462 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:16.772615 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:16.772624 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:16.774717 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:16.774735 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:16.775640 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:16.775649 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:16.775970 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:16.776691 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:16.776724 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:16.779756 (+ 3032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:16.782727 (+ 2971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:16.782739 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:16.782758 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:16.783426 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:16.783784 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:16.783792 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:16.783798 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:16.784034 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:16.784534 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:16.784546 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:16.785000 (+ 454us) client_negotiation.cc:769] Sending connection context
0925 02:57:16.785292 (+ 292us) client_negotiation.cc:240] Negotiation successful
0925 02:57:16.785883 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":272,"threads_started":1}
I20260925 02:57:16.788949 25225 heartbeater.cc:347] Connected to a master server at 127.18.196.254:41505
I20260925 02:57:16.789300 25225 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:16.790211 25225 heartbeater.cc:511] Master 127.18.196.254:41505 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.18.196.195@KRBTEST.COM; defaulting to no policy
I20260925 02:57:16.791877 24902 ts_manager.cc:194] Registered new tserver with Master: e4701f0ee9c0441fb467fd4084b3e378 (127.18.196.194:34611)
I20260925 02:57:16.792757 24902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.194@KRBTEST.COM'} at 127.18.196.194:50965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
I20260925 02:57:16.817464 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:41505
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:17.133078 25231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:17.133338 25231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:17.133440 25231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:17.140837 25231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:17.140993 25231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:17.141175 25231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 02:57:17.150925 25231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:41505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:17.152086 25231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:17.153344 25231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:17.166309 25237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:17.166353 25239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:17.166788 25236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:17.167685 25231 server_base.cc:1034] running on GCE node
I20260925 02:57:17.168308 25231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:17.169258 25231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:17.170492 25231 hybrid_clock.cc:648] HybridClock initialized: now 1790305037170442 us; error 63 us; skew 500 ppm
I20260925 02:57:17.170809 25231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:17 dist-test-slave-8zvk krb5kdc[24857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305037, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:17.177646 25231 init.cc:377] Logged in from keytab as kudu/127.18.196.195@KRBTEST.COM (short username kudu)
I20260925 02:57:17.179147 25231 webserver.cc:460] Webserver started at http://127.18.196.195:45419/ using document root <none> and password file <none>
I20260925 02:57:17.179809 25231 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:17.179916 25231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:17.180168 25231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:17.183092 25231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "1df5d0988caa407786f91e8ff222738d"
format_stamp: "Formatted at 2026-09-25 02:57:17 on dist-test-slave-8zvk"
server_key: "8c92c5d9f737564d9e7aecc7bf29cca6"
server_key_iv: "55af62ae236ceba4ea6587b6729f1302"
server_key_version: "encryptionkey@0"
I20260925 02:57:17.183835 25231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "1df5d0988caa407786f91e8ff222738d"
format_stamp: "Formatted at 2026-09-25 02:57:17 on dist-test-slave-8zvk"
server_key: "8c92c5d9f737564d9e7aecc7bf29cca6"
server_key_iv: "55af62ae236ceba4ea6587b6729f1302"
server_key_version: "encryptionkey@0"
I20260925 02:57:17.188747 25231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:57:17.192106 25246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:17.193429 25231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:57:17.193552 25231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "1df5d0988caa407786f91e8ff222738d"
format_stamp: "Formatted at 2026-09-25 02:57:17 on dist-test-slave-8zvk"
server_key: "8c92c5d9f737564d9e7aecc7bf29cca6"
server_key_iv: "55af62ae236ceba4ea6587b6729f1302"
server_key_version: "encryptionkey@0"
I20260925 02:57:17.193719 25231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:17.198990 25091 heartbeater.cc:503] Master 127.18.196.254:41505 was elected leader, sending a full tablet report...
I20260925 02:57:17.221073 25231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:17.225495 25231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:17.225744 25231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:17.226660 25231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:17.228008 25231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:17.228082 25231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:17.228166 25231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:17.228209 25231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:17.261926 25231 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:39873
I20260925 02:57:17.262029 25358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:39873 every 8 connection(s)
I20260925 02:57:17.263226 25231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 02:57:17.266543 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 25231
I20260925 02:57:17.266726 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 02:57:17.267242 19219 external_mini_cluster.cc:1428] Setting key a6b8eff3dd1d7c67b450c6ed9503e68c
Sep 25 02:57:17 dist-test-slave-8zvk krb5kdc[24857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305037, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:57:17.283406 25227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:17.265727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:45607 (local address 127.18.196.254:41505)
0925 02:57:17.265902 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:17.265908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:17.267326 (+ 1418us) server_negotiation.cc:407] Connection header received
0925 02:57:17.269709 (+ 2383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:17.269713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:17.269797 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:17.269875 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:17.271526 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:17.272271 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:17.273747 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:17.274124 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:17.274213 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:17.277636 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:17.277641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:17.277644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:17.277675 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:17.280179 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:17.280884 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:17.280890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:17.280893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:17.281045 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:17.281600 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:17.281606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:17.281611 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:17.281988 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:17.282154 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:17.282993 (+ 839us) server_negotiation.cc:299] Negotiation successful
0925 02:57:17.283166 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:57:17.284373 25361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:17.266274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:41505 (local address 127.18.196.195:45607)
0925 02:57:17.267131 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:17.267184 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:17.269425 (+ 2241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:17.270180 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:17.270195 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:17.270758 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:17.271371 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:17.271389 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:17.272406 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:17.272410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:17.273625 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:17.273637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:17.274331 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:17.274336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:17.274535 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:17.275185 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:17.275207 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:17.277452 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:17.280301 (+ 2849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:17.280307 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:17.280317 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:17.280752 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:17.281183 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:17.281186 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:17.281188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:17.281451 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:17.282150 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:17.282159 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:17.282555 (+ 396us) client_negotiation.cc:769] Sending connection context
0925 02:57:17.282884 (+ 329us) client_negotiation.cc:240] Negotiation successful
0925 02:57:17.283260 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":348,"threads_started":1}
I20260925 02:57:17.285650 25359 heartbeater.cc:347] Connected to a master server at 127.18.196.254:41505
I20260925 02:57:17.285957 25359 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:17.286635 25359 heartbeater.cc:511] Master 127.18.196.254:41505 requested a full tablet report, sending...
I20260925 02:57:17.288106 24902 ts_manager.cc:194] Registered new tserver with Master: 1df5d0988caa407786f91e8ff222738d (127.18.196.195:39873)
I20260925 02:57:17.288903 24902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.195@KRBTEST.COM'} at 127.18.196.195:45607
I20260925 02:57:17.294620 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/test-admin.keytab.
Sep 25 02:57:17 dist-test-slave-8zvk krb5kdc[24857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:17.322507 19219 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 02:57:17.327764 25366 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:17.325266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55600 (local address 127.18.196.193:38167)
0925 02:57:17.325882 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:17.325890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:17.325920 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:57:17.326062 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:17.326069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:17.326277 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:17.326436 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:17.327301 (+ 865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55600: BlockingRecv error: recv got EOF from 127.0.0.1:55600 (error 108)
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":281,"threads_started":1}
W20260925 02:57:17.327792 19219 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/krb5cc' of type FILE
Sep 25 02:57:17 dist-test-slave-8zvk krb5kdc[24857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:17.329406 19219 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 02:57:17 dist-test-slave-8zvk krb5kdc[24857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:17.330899 19219 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 02:57:17 dist-test-slave-8zvk krb5kdc[24857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.193@KRBTEST.COM
I20260925 02:57:17.347015 25366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:17.332612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55616 (local address 127.18.196.193:38167)
0925 02:57:17.332824 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:17.332831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:17.332999 (+ 168us) server_negotiation.cc:407] Connection header received
0925 02:57:17.333127 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:17.333131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:17.333194 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:17.333280 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:17.334421 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:17.335595 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:17.336810 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:17.337388 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:17.337524 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:17.341280 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:17.341290 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:17.341304 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:17.341353 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:17.344211 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:17.344971 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:17.344978 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:17.344980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:17.345096 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:17.345479 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:17.345483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:17.345492 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:17.345875 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:17.346103 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:17.346451 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 02:57:17.346645 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260925 02:57:17.352787 25366 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:17.350970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55622 (local address 127.18.196.193:38167)
0925 02:57:17.351155 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:17.351164 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:17.351452 (+ 288us) server_negotiation.cc:407] Connection header received
0925 02:57:17.351646 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:17.351649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:17.351704 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:17.351800 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:17.352509 (+ 709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55622: BlockingRecv error: recv got EOF from 127.0.0.1:55622 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 02:57:17.352941 19219 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/krb5cc' of type FILE
Sep 25 02:57:17 dist-test-slave-8zvk krb5kdc[24857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:17.354508 19219 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 02:57:17 dist-test-slave-8zvk krb5kdc[24857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:17.355947 19219 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 02:57:17 dist-test-slave-8zvk krb5kdc[24857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.193@KRBTEST.COM
I20260925 02:57:17.371040 25366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:17.357394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55624 (local address 127.18.196.193:38167)
0925 02:57:17.357609 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:17.357615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:17.357783 (+ 168us) server_negotiation.cc:407] Connection header received
0925 02:57:17.357902 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:17.357906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:17.357964 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:17.358052 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:17.359050 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:17.359774 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:17.361282 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:17.361760 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:17.361881 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:17.365530 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:17.365536 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:17.365539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:17.365587 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:17.368473 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:17.369101 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:17.369105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:17.369107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:17.369236 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:17.369611 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:17.369615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:17.369618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:17.369920 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:17.370140 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:17.370434 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 02:57:17.370650 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260925 02:57:17.375579 25366 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:17.373707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55628 (local address 127.18.196.193:38167)
0925 02:57:17.373937 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:17.373943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:17.374229 (+ 286us) server_negotiation.cc:407] Connection header received
0925 02:57:17.374342 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:17.374348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:17.374417 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:17.374499 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:17.375361 (+ 862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55628: BlockingRecv error: recv got EOF from 127.0.0.1:55628 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 02:57:17.375756 19219 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304987644417-19219-0/krb5kdc/krb5cc' of type FILE
Sep 25 02:57:17 dist-test-slave-8zvk krb5kdc[24857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:17.377300 19219 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 02:57:17 dist-test-slave-8zvk krb5kdc[24857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:17.378588 19219 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 02:57:17 dist-test-slave-8zvk krb5kdc[24857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.193@KRBTEST.COM
I20260925 02:57:17.393273 25366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:17.380020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55634 (local address 127.18.196.193:38167)
0925 02:57:17.380225 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:17.380231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:17.380474 (+ 243us) server_negotiation.cc:407] Connection header received
0925 02:57:17.380562 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:17.380565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:17.380619 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:17.380696 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:17.381620 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:17.382412 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:17.383737 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:17.384184 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:17.384287 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:17.387730 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:17.387735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:17.387738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:17.387770 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:17.390318 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:17.391130 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:17.391137 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:17.391141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:17.391295 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:17.391733 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:17.391741 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:17.391749 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:17.392072 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:17.392424 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:17.392718 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 02:57:17.392963 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:57:17.394789 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 24963
I20260925 02:57:17.407614 25366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:17.395483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55648 (local address 127.18.196.193:38167)
0925 02:57:17.395658 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:17.395663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:17.395984 (+ 321us) server_negotiation.cc:407] Connection header received
0925 02:57:17.396104 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:17.396107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:17.396159 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:17.396233 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:17.397189 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:17.398252 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:17.399587 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:17.400205 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:17.400320 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:17.402491 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:17.402497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:17.402504 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:17.402539 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:17.405067 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:17.405697 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:17.405702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:17.405706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:17.405827 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:17.406217 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:17.406222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:17.406226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:17.406484 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:17.406757 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:17.407124 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 02:57:17.407325 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 02:57:17.408191 25086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:17.492347 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 24963
I20260925 02:57:17.509928 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 25097
Sep 25 02:57:17 dist-test-slave-8zvk krb5kdc[24857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.194@KRBTEST.COM
I20260925 02:57:17.525496 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:17.510805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.18.196.194:34611)
0925 02:57:17.511446 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:17.511453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:17.511479 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:57:17.511539 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:17.511545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:17.511750 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:17.511888 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:17.512832 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:17.513961 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:17.515299 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:17.515844 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:17.515963 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:17.519727 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:17.519738 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:17.519751 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:17.519791 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:17.522667 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:17.523415 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:17.523421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:17.523424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:17.523564 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:17.523956 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:17.523961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:17.523963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:17.524291 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:17.524539 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:17.524916 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 02:57:17.525205 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":359,"threads_started":1}
I20260925 02:57:17.526734 25220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:17.607934 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 25097
I20260925 02:57:17.624035 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 25231
Sep 25 02:57:17 dist-test-slave-8zvk krb5kdc[24857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.195@KRBTEST.COM
I20260925 02:57:17.639930 25381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:17.625099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47336 (local address 127.18.196.195:39873)
0925 02:57:17.625739 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:17.625746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:17.625774 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:57:17.625835 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:17.625841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:17.626063 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:17.626234 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:17.627361 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:17.628375 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:17.629610 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:17.630172 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:17.630313 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:17.634323 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:17.634337 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:17.634354 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:17.634400 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:17.637256 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:17.637882 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:17.637889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:17.637891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:17.638006 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:17.638315 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:17.638318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:17.638320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:17.638651 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:17.638945 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:17.639336 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 02:57:17.639648 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":350,"threads_started":1}
I20260925 02:57:17.641259 25354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:17.723387 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 25231
I20260925 02:57:17.739948 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 24870
I20260925 02:57:17.741210 24932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:17.846733 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 24870
2026-09-25T02:57:17Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4163 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:57:17 dist-test-slave-8zvk krb5kdc[25386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:57:17 dist-test-slave-8zvk krb5kdc[25386](info): set up 2 sockets
Sep 25 02:57:17 dist-test-slave-8zvk krb5kdc[25386](info): commencing operation
krb5kdc: starting...
W20260925 02:57:18.918895 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 02:57:19 dist-test-slave-8zvk krb5kdc[25386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:57:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:57:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.254.keytab.
Entry for principal oryx/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.254.keytab.
I20260925 02:57:19.124346 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:40775
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.254.keytab
--principal=oryx/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:40775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:19.441708 25399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:19.441977 25399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:19.442066 25399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:19.449092 25399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:19.449188 25399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:19.449234 25399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:19.449292 25399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:19.449337 25399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:19.458801 25399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:40775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.254.keytab
--principal=oryx/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:40775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:19.459991 25399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:19.461606 25399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:19.472882 25405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:19.473208 25404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:19.473275 25407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:19.474045 25399 server_base.cc:1034] running on GCE node
I20260925 02:57:19.474771 25399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:19.476078 25399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:19.477277 25399 hybrid_clock.cc:648] HybridClock initialized: now 1790305039477242 us; error 32 us; skew 500 ppm
I20260925 02:57:19.477615 25399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:19 dist-test-slave-8zvk krb5kdc[25386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305039, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:19.480604 25399 init.cc:377] Logged in from keytab as oryx/127.18.196.254@KRBTEST.COM (short username oryx)
I20260925 02:57:19.481956 25399 webserver.cc:460] Webserver started at http://127.18.196.254:34485/ using document root <none> and password file <none>
I20260925 02:57:19.482597 25399 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:19.482728 25399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:19.483052 25399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:19.485845 25399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "d736dfd4e08d41148d0bbc56465191ca"
format_stamp: "Formatted at 2026-09-25 02:57:19 on dist-test-slave-8zvk"
server_key: "b6aa53fd59d5b1c091faaefc0c1be9e6"
server_key_iv: "3bc671c29dfe2f89aa5c0ca79075bd8f"
server_key_version: "encryptionkey@0"
I20260925 02:57:19.486598 25399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "d736dfd4e08d41148d0bbc56465191ca"
format_stamp: "Formatted at 2026-09-25 02:57:19 on dist-test-slave-8zvk"
server_key: "b6aa53fd59d5b1c091faaefc0c1be9e6"
server_key_iv: "3bc671c29dfe2f89aa5c0ca79075bd8f"
server_key_version: "encryptionkey@0"
I20260925 02:57:19.491488 25399 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:57:19.494752 25414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:19.496019 25399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:57:19.496200 25399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "d736dfd4e08d41148d0bbc56465191ca"
format_stamp: "Formatted at 2026-09-25 02:57:19 on dist-test-slave-8zvk"
server_key: "b6aa53fd59d5b1c091faaefc0c1be9e6"
server_key_iv: "3bc671c29dfe2f89aa5c0ca79075bd8f"
server_key_version: "encryptionkey@0"
I20260925 02:57:19.496398 25399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:19.512742 25399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:19.516793 25399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:19.517077 25399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:19.534368 25399 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:40775
I20260925 02:57:19.534415 25465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:40775 every 8 connection(s)
I20260925 02:57:19.535719 25399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:57:19.539412 25466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:19.542613 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 25399
I20260925 02:57:19.542845 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:57:19.543792 19219 external_mini_cluster.cc:1428] Setting key 9c8079d773ff9beabbd084d62631c3cc
I20260925 02:57:19.552302 25466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca: Bootstrap starting.
I20260925 02:57:19.555974 25466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:19.557314 25466 log.cc:824] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:57:19 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM
I20260925 02:57:19.560398 25466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca: No bootstrap required, opened a new log
I20260925 02:57:19.567063 25469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:19.545868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56658 (local address 127.18.196.254:40775)
0925 02:57:19.546615 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:19.546627 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:19.546664 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:57:19.548292 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:19.548314 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:19.548687 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:19.549009 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:19.550819 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:19.552199 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:19.553942 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:19.554618 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:19.554753 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:19.559673 (+ 4920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:19.559683 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:19.559700 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:19.559739 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:19.563033 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:19.563880 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:19.563889 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:19.563894 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:19.564063 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:19.564486 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:19.564489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:19.564491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:19.564979 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:19.565259 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:19.565643 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 02:57:19.565988 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":356,"threads_started":1}
I20260925 02:57:19.569214 25466 raft_consensus.cc:348] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } }
I20260925 02:57:19.569567 25466 raft_consensus.cc:374] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:19.569675 25466 raft_consensus.cc:729] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d736dfd4e08d41148d0bbc56465191ca, State: Initialized, Role: FOLLOWER
I20260925 02:57:19.570209 25466 consensus_queue.cc:260] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } }
I20260925 02:57:19.570412 25466 raft_consensus.cc:388] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:57:19.570547 25466 raft_consensus.cc:482] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:57:19.570763 25466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:19.573030 25466 raft_consensus.cc:504] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } }
I20260925 02:57:19.573526 25466 leader_election.cc:302] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d736dfd4e08d41148d0bbc56465191ca; no voters:
I20260925 02:57:19.574051 25466 leader_election.cc:288] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:57:19.574179 25471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:19.574870 25471 raft_consensus.cc:686] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [term 1 LEADER]: Becoming Leader. State: Replica: d736dfd4e08d41148d0bbc56465191ca, State: Running, Role: LEADER
I20260925 02:57:19.575450 25471 consensus_queue.cc:237] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } }
I20260925 02:57:19.576018 25466 sys_catalog.cc:564] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:19.578408 25473 sys_catalog.cc:455] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader d736dfd4e08d41148d0bbc56465191ca. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } }
I20260925 02:57:19.578804 25473 sys_catalog.cc:458] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:19.579312 25472 sys_catalog.cc:455] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } }
I20260925 02:57:19.580003 25472 sys_catalog.cc:458] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:19.580421 25480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:19.585314 25480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:19.593669 25480 catalog_manager.cc:1347] Generated new cluster ID: 81e88c27e36e498f87565d50facd8f55
I20260925 02:57:19.593787 25480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:19.604857 25480 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:57:19.606017 25480 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:19.613408 25480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca: Generated new TSK 0
I20260925 02:57:19.614090 25480 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.18.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.193.keytab.
Entry for principal oryx/127.18.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.193.keytab.
I20260925 02:57:19.670132 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:40775
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.193.keytab
--principal=oryx/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:19.992616 25492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:19.992872 25492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:19.992982 25492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:19.999826 25492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:19.999939 25492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:20.000108 25492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 02:57:20.009805 25492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.193.keytab
--principal=oryx/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:40775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:20.010965 25492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:20.012177 25492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:20.024221 25497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:20.024305 25498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:20.024273 25500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:20.024822 25492 server_base.cc:1034] running on GCE node
I20260925 02:57:20.025440 25492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:20.026252 25492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:20.027442 25492 hybrid_clock.cc:648] HybridClock initialized: now 1790305040027402 us; error 34 us; skew 500 ppm
I20260925 02:57:20.027705 25492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:20 dist-test-slave-8zvk krb5kdc[25386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305040, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:20.030575 25492 init.cc:377] Logged in from keytab as oryx/127.18.196.193@KRBTEST.COM (short username oryx)
I20260925 02:57:20.031812 25492 webserver.cc:460] Webserver started at http://127.18.196.193:42395/ using document root <none> and password file <none>
I20260925 02:57:20.032377 25492 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:20.032523 25492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:20.032793 25492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:20.035410 25492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "c2fa8d55ea204f48a127e243690cb618"
format_stamp: "Formatted at 2026-09-25 02:57:20 on dist-test-slave-8zvk"
server_key: "97f72b910581987a31feebcc5be1c759"
server_key_iv: "63dc6cba9ef20b973a81e89196e15add"
server_key_version: "encryptionkey@0"
I20260925 02:57:20.036140 25492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "c2fa8d55ea204f48a127e243690cb618"
format_stamp: "Formatted at 2026-09-25 02:57:20 on dist-test-slave-8zvk"
server_key: "97f72b910581987a31feebcc5be1c759"
server_key_iv: "63dc6cba9ef20b973a81e89196e15add"
server_key_version: "encryptionkey@0"
I20260925 02:57:20.040606 25492 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:57:20.043650 25507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:20.044694 25492 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:57:20.044831 25492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "c2fa8d55ea204f48a127e243690cb618"
format_stamp: "Formatted at 2026-09-25 02:57:20 on dist-test-slave-8zvk"
server_key: "97f72b910581987a31feebcc5be1c759"
server_key_iv: "63dc6cba9ef20b973a81e89196e15add"
server_key_version: "encryptionkey@0"
I20260925 02:57:20.045002 25492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:20.073145 25492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:20.076628 25492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:20.076869 25492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:20.077680 25492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:20.078871 25492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:20.078951 25492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:20.079088 25492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:20.079149 25492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:20.110306 25492 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:38247
I20260925 02:57:20.110342 25619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:38247 every 8 connection(s)
I20260925 02:57:20.111569 25492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 02:57:20.121629 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 25492
I20260925 02:57:20.121830 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 02:57:20.122623 19219 external_mini_cluster.cc:1428] Setting key bddd01bb2fabb2501bd4c1e671cbed73
Sep 25 02:57:20 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305040, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM
I20260925 02:57:20.131161 25622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:20.113874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:39183 (local address 127.18.196.254:40775)
0925 02:57:20.114250 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:20.114254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:20.115287 (+ 1033us) server_negotiation.cc:407] Connection header received
0925 02:57:20.117038 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:20.117044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:20.117155 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:20.117272 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:20.118650 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:20.119420 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:20.120940 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:20.121326 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:20.121477 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:20.125267 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:20.125272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:20.125275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:20.125306 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:20.127857 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:20.128767 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:20.128771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:20.128773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:20.128896 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:20.129411 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:20.129414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:20.129416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:20.129666 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:20.129773 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:20.130585 (+ 812us) server_negotiation.cc:299] Negotiation successful
0925 02:57:20.130754 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":197,"threads_started":1}
I20260925 02:57:20.132053 25623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:20.114411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.18.196.193:39183)
0925 02:57:20.115122 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:20.115176 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:20.116774 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:20.117492 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:20.117500 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:20.117962 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:20.118478 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:20.118494 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:20.119596 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:20.119600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:20.120773 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:20.120785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:20.121580 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:20.121587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:20.121859 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:20.122549 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:20.122578 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:20.125022 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:20.128007 (+ 2985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:20.128018 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:20.128035 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:20.128654 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:20.129005 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:20.129012 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:20.129019 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:20.129287 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:20.129790 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:20.129801 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:20.130237 (+ 436us) client_negotiation.cc:769] Sending connection context
0925 02:57:20.130524 (+ 287us) client_negotiation.cc:240] Negotiation successful
0925 02:57:20.131033 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":266,"threads_started":1}
I20260925 02:57:20.133523 25620 heartbeater.cc:347] Connected to a master server at 127.18.196.254:40775
I20260925 02:57:20.133846 25620 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:20.134550 25620 heartbeater.cc:511] Master 127.18.196.254:40775 requested a full tablet report, sending...
I20260925 02:57:20.136538 25431 ts_manager.cc:194] Registered new tserver with Master: c2fa8d55ea204f48a127e243690cb618 (127.18.196.193:38247)
I20260925 02:57:20.137928 25431 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.196.193@KRBTEST.COM'} at 127.18.196.193:39183
WARNING: no policy specified for oryx/127.18.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.194.keytab.
Entry for principal oryx/127.18.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.194.keytab.
I20260925 02:57:20.169811 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:40775
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.194.keytab
--principal=oryx/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:20.485846 25626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:20.486123 25626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:20.486275 25626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:20.493894 25626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:20.494040 25626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:20.494225 25626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 02:57:20.504496 25626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.194.keytab
--principal=oryx/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:40775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:20.505808 25626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:20.507207 25626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:20.520493 25634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:20.520452 25632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:20.520532 25631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:20.521308 25626 server_base.cc:1034] running on GCE node
I20260925 02:57:20.521991 25626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:20.522970 25626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:20.524214 25626 hybrid_clock.cc:648] HybridClock initialized: now 1790305040524162 us; error 38 us; skew 500 ppm
I20260925 02:57:20.524538 25626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:20 dist-test-slave-8zvk krb5kdc[25386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305040, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:20.527734 25626 init.cc:377] Logged in from keytab as oryx/127.18.196.194@KRBTEST.COM (short username oryx)
I20260925 02:57:20.529137 25626 webserver.cc:460] Webserver started at http://127.18.196.194:46411/ using document root <none> and password file <none>
I20260925 02:57:20.529803 25626 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:20.529951 25626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:20.530252 25626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:20.533190 25626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "bb0d89109f23435db76236a5f64b25d5"
format_stamp: "Formatted at 2026-09-25 02:57:20 on dist-test-slave-8zvk"
server_key: "19cc7f7ee7df85391fd716127804dfdc"
server_key_iv: "970a24ac0927882dc0bad979a20dcfac"
server_key_version: "encryptionkey@0"
I20260925 02:57:20.534024 25626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "bb0d89109f23435db76236a5f64b25d5"
format_stamp: "Formatted at 2026-09-25 02:57:20 on dist-test-slave-8zvk"
server_key: "19cc7f7ee7df85391fd716127804dfdc"
server_key_iv: "970a24ac0927882dc0bad979a20dcfac"
server_key_version: "encryptionkey@0"
I20260925 02:57:20.539316 25626 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 02:57:20.542789 25641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:20.544302 25626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:57:20.544493 25626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "bb0d89109f23435db76236a5f64b25d5"
format_stamp: "Formatted at 2026-09-25 02:57:20 on dist-test-slave-8zvk"
server_key: "19cc7f7ee7df85391fd716127804dfdc"
server_key_iv: "970a24ac0927882dc0bad979a20dcfac"
server_key_version: "encryptionkey@0"
I20260925 02:57:20.544699 25626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:20.555866 25626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:20.559823 25626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:20.560112 25626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:20.561028 25626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:20.562330 25626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:20.562448 25626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:20.562590 25626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:20.562665 25626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:20.595865 25626 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:36001
I20260925 02:57:20.595918 25753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:36001 every 8 connection(s)
I20260925 02:57:20.597119 25626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 02:57:20.598769 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 25626
I20260925 02:57:20.598979 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 02:57:20.599594 19219 external_mini_cluster.cc:1428] Setting key 33e65554cdf5af1335fd3c38522ef5f6
Sep 25 02:57:20 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305040, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM
I20260925 02:57:20.617631 25622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:20.599788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:34349 (local address 127.18.196.254:40775)
0925 02:57:20.600011 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:20.600015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:20.601312 (+ 1297us) server_negotiation.cc:407] Connection header received
0925 02:57:20.603249 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:20.603254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:20.603343 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:20.603451 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:20.604869 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:20.605587 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:20.607155 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:20.607877 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:20.608017 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:20.611615 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:20.611625 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:20.611632 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:20.611675 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:20.614571 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:20.615363 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:20.615367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:20.615370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:20.615514 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:20.615948 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:20.615959 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:20.615961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:20.616231 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:20.616398 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:20.617036 (+ 638us) server_negotiation.cc:299] Negotiation successful
0925 02:57:20.617256 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:57:20.618533 25756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:20.600393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.18.196.194:34349)
0925 02:57:20.601172 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:20.601212 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:20.602993 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:20.603678 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:20.603686 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:20.604156 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:20.604705 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:20.604721 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:20.605744 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:20.605748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:20.607011 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:20.607023 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:20.608115 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:20.608120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:20.608317 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:20.608986 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:20.609007 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:20.611385 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:20.614709 (+ 3324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:20.614717 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:20.614731 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:20.615227 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:20.615630 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:20.615635 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:20.615638 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:20.615823 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:20.616380 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:20.616387 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:20.616694 (+ 307us) client_negotiation.cc:769] Sending connection context
0925 02:57:20.616948 (+ 254us) client_negotiation.cc:240] Negotiation successful
0925 02:57:20.617453 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":300,"threads_started":1}
I20260925 02:57:20.620038 25754 heartbeater.cc:347] Connected to a master server at 127.18.196.254:40775
I20260925 02:57:20.620316 25754 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.18.196.195@KRBTEST.COM; defaulting to no policy
I20260925 02:57:20.621032 25754 heartbeater.cc:511] Master 127.18.196.254:40775 requested a full tablet report, sending...
I20260925 02:57:20.622368 25431 ts_manager.cc:194] Registered new tserver with Master: bb0d89109f23435db76236a5f64b25d5 (127.18.196.194:36001)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.196.195@KRBTEST.COM" created.
I20260925 02:57:20.623521 25431 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.196.194@KRBTEST.COM'} at 127.18.196.194:34349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.195.keytab.
Entry for principal oryx/127.18.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.195.keytab.
I20260925 02:57:20.647675 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:40775
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.195.keytab
--principal=oryx/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:20.961687 25759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:20.961933 25759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:20.962090 25759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:20.968837 25759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:20.968924 25759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:20.969080 25759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 02:57:20.978322 25759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.195.keytab
--principal=oryx/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:40775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:20.979429 25759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:20.980659 25759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:20.992484 25764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:20.992551 25765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:20.992597 25767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:20.993302 25759 server_base.cc:1034] running on GCE node
I20260925 02:57:20.993916 25759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:20.994741 25759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:20.995957 25759 hybrid_clock.cc:648] HybridClock initialized: now 1790305040995934 us; error 52 us; skew 500 ppm
I20260925 02:57:20.996246 25759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:20 dist-test-slave-8zvk krb5kdc[25386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305040, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:20.999102 25759 init.cc:377] Logged in from keytab as oryx/127.18.196.195@KRBTEST.COM (short username oryx)
I20260925 02:57:21.000553 25759 webserver.cc:460] Webserver started at http://127.18.196.195:35019/ using document root <none> and password file <none>
I20260925 02:57:21.001204 25759 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:21.001341 25759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:21.001653 25759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:21.004379 25759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "920f7345bb8940ff98552b67ad8fb652"
format_stamp: "Formatted at 2026-09-25 02:57:20 on dist-test-slave-8zvk"
server_key: "2e7dc84cf5c345c70ad755c62a25f687"
server_key_iv: "41c4ee872420e5fbf72cf58b5b8da6a1"
server_key_version: "encryptionkey@0"
I20260925 02:57:21.005162 25759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "920f7345bb8940ff98552b67ad8fb652"
format_stamp: "Formatted at 2026-09-25 02:57:20 on dist-test-slave-8zvk"
server_key: "2e7dc84cf5c345c70ad755c62a25f687"
server_key_iv: "41c4ee872420e5fbf72cf58b5b8da6a1"
server_key_version: "encryptionkey@0"
I20260925 02:57:21.009644 25759 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 02:57:21.012681 25774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:21.013867 25759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:57:21.014032 25759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "920f7345bb8940ff98552b67ad8fb652"
format_stamp: "Formatted at 2026-09-25 02:57:20 on dist-test-slave-8zvk"
server_key: "2e7dc84cf5c345c70ad755c62a25f687"
server_key_iv: "41c4ee872420e5fbf72cf58b5b8da6a1"
server_key_version: "encryptionkey@0"
I20260925 02:57:21.014241 25759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:21.033603 25759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:21.037141 25759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:21.037403 25759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:21.038170 25759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:21.039307 25759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:21.039399 25759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:21.039534 25759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:21.039597 25759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:21.072029 25759 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:43605
I20260925 02:57:21.072109 25886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:43605 every 8 connection(s)
I20260925 02:57:21.073220 25759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 02:57:21.075718 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 25759
I20260925 02:57:21.075910 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 02:57:21.076467 19219 external_mini_cluster.cc:1428] Setting key 0457e266dfe96fed20fd7fec000fdcad
Sep 25 02:57:21 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305040, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM
I20260925 02:57:21.093462 25622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:21.075964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:48007 (local address 127.18.196.254:40775)
0925 02:57:21.076171 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:21.076179 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:21.077696 (+ 1517us) server_negotiation.cc:407] Connection header received
0925 02:57:21.079862 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:21.079866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:21.079957 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:21.080057 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:21.081443 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.082203 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.083842 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.084282 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.084385 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:21.087699 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:21.087704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:21.087707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:21.087738 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:21.090397 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:21.091209 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:21.091214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:21.091216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:21.091356 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:21.091790 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:21.091794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:21.091796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:21.092066 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:21.092229 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:21.092892 (+ 663us) server_negotiation.cc:299] Negotiation successful
0925 02:57:21.093101 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:57:21.094365 25889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:21.076455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.18.196.195:48007)
0925 02:57:21.077350 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:21.077402 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:21.079549 (+ 2147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:21.080239 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:21.080247 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:21.080709 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:21.081241 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:21.081259 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.082334 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.082337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:21.083700 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:21.083713 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.084467 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.084472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:21.084660 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:21.085249 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:21.085269 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:21.087466 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:21.090554 (+ 3088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:21.090563 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:21.090577 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:21.091076 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:21.091472 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:21.091477 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:21.091481 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:21.091659 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:21.092213 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:21.092221 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:21.092530 (+ 309us) client_negotiation.cc:769] Sending connection context
0925 02:57:21.092787 (+ 257us) client_negotiation.cc:240] Negotiation successful
0925 02:57:21.093292 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":387,"threads_started":1}
I20260925 02:57:21.095642 25887 heartbeater.cc:347] Connected to a master server at 127.18.196.254:40775
I20260925 02:57:21.095942 25887 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:21.096638 25887 heartbeater.cc:511] Master 127.18.196.254:40775 requested a full tablet report, sending...
I20260925 02:57:21.097970 25431 ts_manager.cc:194] Registered new tserver with Master: 920f7345bb8940ff98552b67ad8fb652 (127.18.196.195:43605)
I20260925 02:57:21.098873 25431 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.196.195@KRBTEST.COM'} at 127.18.196.195:48007
I20260925 02:57:21.103732 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.18.196.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.196.253@KRBTEST.COM" created.
I20260925 02:57:21.140888 25620 heartbeater.cc:503] Master 127.18.196.254:40775 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.196.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.253.keytab.
Entry for principal oryx/127.18.196.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.253.keytab.
I20260925 02:57:21.150619 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.253:44133
--webserver_interface=127.18.196.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.253.keytab
--principal=oryx/127.18.196.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:40775,127.18.196.253:44133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:21.466259 25892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:21.466585 25892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:21.466755 25892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:21.474030 25892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:21.474174 25892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:21.474274 25892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:21.474380 25892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:21.474495 25892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:21.484485 25892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:40775,127.18.196.253:44133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.253.keytab
--principal=oryx/127.18.196.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.253:44133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.196.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
W20260925 02:57:21.484762 25892 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.18.196.254:40775,127.18.196.253:44133'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 02:57:21.485682 25892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:21.486958 25892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:21.498283 25897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:21.498376 25898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:21.498533 25900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:21.499321 25892 server_base.cc:1034] running on GCE node
I20260925 02:57:21.500020 25892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:21.501364 25892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:21.502650 25892 hybrid_clock.cc:648] HybridClock initialized: now 1790305041502572 us; error 57 us; skew 500 ppm
I20260925 02:57:21.503028 25892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:21 dist-test-slave-8zvk krb5kdc[25386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305041, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:21.505880 25892 init.cc:377] Logged in from keytab as oryx/127.18.196.253@KRBTEST.COM (short username oryx)
I20260925 02:57:21.507268 25892 webserver.cc:460] Webserver started at http://127.18.196.253:42479/ using document root <none> and password file <none>
I20260925 02:57:21.507887 25892 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:21.508018 25892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:21.508308 25892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:21.511140 25892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/data/instance:
uuid: "ab6a987abd184695be36cdde2d06d24e"
format_stamp: "Formatted at 2026-09-25 02:57:21 on dist-test-slave-8zvk"
server_key: "7aa926b9a864a17c621becc9bcf34873"
server_key_iv: "597f0104855a5d613bdd81ecacba10ae"
server_key_version: "encryptionkey@0"
I20260925 02:57:21.511902 25892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/wal/instance:
uuid: "ab6a987abd184695be36cdde2d06d24e"
format_stamp: "Formatted at 2026-09-25 02:57:21 on dist-test-slave-8zvk"
server_key: "7aa926b9a864a17c621becc9bcf34873"
server_key_iv: "597f0104855a5d613bdd81ecacba10ae"
server_key_version: "encryptionkey@0"
I20260925 02:57:21.516866 25892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:57:21.520005 25907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:21.521216 25892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:57:21.521395 25892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/wal
uuid: "ab6a987abd184695be36cdde2d06d24e"
format_stamp: "Formatted at 2026-09-25 02:57:21 on dist-test-slave-8zvk"
server_key: "7aa926b9a864a17c621becc9bcf34873"
server_key_iv: "597f0104855a5d613bdd81ecacba10ae"
server_key_version: "encryptionkey@0"
I20260925 02:57:21.521602 25892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:21.535669 25892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:21.540433 25892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:21.540776 25892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:21.559432 25892 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.253:44133
I20260925 02:57:21.559470 25958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.253:44133 every 8 connection(s)
I20260925 02:57:21.560997 25892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/data/info.pb
I20260925 02:57:21.564483 25959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:21.568893 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 25892
I20260925 02:57:21.571609 25959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:57:21 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.196.253@KRBTEST.COM
Sep 25 02:57:21 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305041, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.253@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM
I20260925 02:57:21.591079 25622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:21.575513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43302 (local address 127.18.196.254:40775)
0925 02:57:21.575825 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:21.575834 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:21.576433 (+ 599us) server_negotiation.cc:407] Connection header received
0925 02:57:21.576641 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:21.576652 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:21.576713 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:21.576813 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:21.578003 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.578794 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.580819 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.581241 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.581390 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:21.584997 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:21.585002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:21.585005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:21.585036 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:21.588151 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:21.588939 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:21.588943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:21.588945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:21.589072 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:21.589460 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:21.589463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:21.589465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:21.589864 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:21.590010 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:21.590608 (+ 598us) server_negotiation.cc:299] Negotiation successful
0925 02:57:21.590766 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 02:57:21.592089 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:21.571047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47232 (local address 127.18.196.253:44133)
0925 02:57:21.571796 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:21.571810 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:21.571865 (+ 55us) server_negotiation.cc:407] Connection header received
0925 02:57:21.573558 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:21.573580 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:21.573963 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:21.574266 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:21.576113 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.577683 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.579053 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.580135 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.580295 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:21.584553 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:21.584563 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:21.584578 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:21.584619 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:21.587423 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:21.588246 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:21.588267 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:21.588275 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:21.588492 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:21.588857 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:21.588864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:21.588870 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:21.589521 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:21.589734 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:21.590244 (+ 510us) server_negotiation.cc:299] Negotiation successful
0925 02:57:21.590688 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":548,"spinlock_wait_cycles":20352,"thread_start_us":333,"threads_started":1}
I20260925 02:57:21.592110 25963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:21.575472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.0.0.1:43302)
0925 02:57:21.576288 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:21.576328 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:21.576511 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:21.576867 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:21.576874 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:21.577508 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:21.577839 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:21.577852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.578977 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.578983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:21.580684 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:21.580695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.581465 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.581472 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:21.581631 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:21.582216 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:21.582235 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:21.584820 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:21.588277 (+ 3457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:21.588285 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:21.588292 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:21.588834 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:21.589200 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:21.589204 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:21.589208 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:21.589356 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:21.590008 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:21.590018 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:21.590238 (+ 220us) client_negotiation.cc:769] Sending connection context
0925 02:57:21.590481 (+ 243us) client_negotiation.cc:240] Negotiation successful
0925 02:57:21.590804 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":79,"spinlock_wait_cycles":21248,"thread_start_us":462,"threads_started":1}
I20260925 02:57:21.594039 25959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:57:21 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305041, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.253@KRBTEST.COM for oryx/127.18.196.253@KRBTEST.COM
I20260925 02:57:21.609567 25963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:21.594658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:44133 (local address 127.0.0.1:47234)
0925 02:57:21.594809 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:21.594828 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:21.594946 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:21.599292 (+ 4346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:21.599297 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:21.599710 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:21.599909 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:21.599926 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.600942 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.600946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:21.601813 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:21.601824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.602442 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.602446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:21.602571 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:21.603114 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:21.603130 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:21.605090 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:21.607787 (+ 2697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:21.607790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:21.607792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:21.608164 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:21.608443 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:21.608446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:21.608448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:21.608597 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:21.608995 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:21.608998 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:21.609093 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 02:57:21.609187 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 02:57:21.609360 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 02:57:21.609750 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:21.594872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.18.196.253:44133)
0925 02:57:21.599019 (+ 4147us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:21.599025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:21.599043 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:57:21.599098 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:21.599102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:21.599153 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:21.599228 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:21.600034 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.600823 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.601931 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.602325 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.602416 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:21.605263 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:21.605268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:21.605271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:21.605298 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:21.607690 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:21.608252 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:21.608255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:21.608257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:21.608361 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:21.608678 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:21.608681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:21.608683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:21.608900 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:21.608995 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:21.609266 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 02:57:21.609456 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":4026}
I20260925 02:57:21.614078 25959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e: Bootstrap starting.
I20260925 02:57:21.617079 25959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:21.618090 25959 log.cc:824] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:21.620638 25959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e: No bootstrap required, opened a new log
I20260925 02:57:21.626590 25754 heartbeater.cc:503] Master 127.18.196.254:40775 was elected leader, sending a full tablet report...
I20260925 02:57:21.627900 25959 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } }
I20260925 02:57:21.628283 25959 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:21.628376 25959 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab6a987abd184695be36cdde2d06d24e, State: Initialized, Role: FOLLOWER
I20260925 02:57:21.628883 25959 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } }
I20260925 02:57:21.629588 25965 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } } }
I20260925 02:57:21.629881 25965 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:57:21.630442 25959 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [sys.catalog]: configured and running, proceeding with master startup.
W20260925 02:57:21.638779 25976 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:57:21.638968 25976 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:57:21.641165 25892 master_runner.cc:428] Detected that this master ab6a987abd184695be36cdde2d06d24e is joining an existing cluster
I20260925 02:57:21.641260 25892 master_runner.cc:432] Initiating AddMaster RPC to add 127.18.196.253:44133
I20260925 02:57:21.662648 25622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:21.645114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.18.196.254:40775)
0925 02:57:21.645336 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:21.645343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:21.645966 (+ 623us) server_negotiation.cc:407] Connection header received
0925 02:57:21.646216 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:21.646222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:21.646298 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:21.646363 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:21.647876 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.648971 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.650996 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.651620 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.651705 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:21.654064 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:21.654071 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:21.654078 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:21.654119 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:21.657635 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:21.658346 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:21.658352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:21.658372 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:21.658564 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:21.659045 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:21.659052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:21.659055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:21.659380 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:21.659503 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:21.661992 (+ 2489us) server_negotiation.cc:299] Negotiation successful
0925 02:57:21.662230 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 02:57:21.663861 25982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:21.645183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.0.0.1:43304)
0925 02:57:21.645864 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:21.645886 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:21.646076 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:21.646592 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:21.646598 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:21.647293 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:21.647697 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:21.647710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.649151 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.649158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:21.650819 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:21.650853 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.651945 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.651951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:21.652104 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:21.652792 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:21.652812 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:21.653854 (+ 1042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:21.657780 (+ 3926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:21.657784 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:21.657789 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:21.658215 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:21.658705 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:21.658710 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:21.658713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:21.658905 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:21.659601 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:21.659608 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:21.659816 (+ 208us) client_negotiation.cc:769] Sending connection context
0925 02:57:21.662917 (+ 3101us) client_negotiation.cc:240] Negotiation successful
0925 02:57:21.663284 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":437,"threads_started":1}
I20260925 02:57:21.673054 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:21.646470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47238 (local address 127.18.196.253:44133)
0925 02:57:21.646623 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:21.646629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:21.658011 (+ 11382us) server_negotiation.cc:407] Connection header received
0925 02:57:21.658304 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:21.658311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:21.658393 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:21.658458 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:21.659772 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.660512 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.661811 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.662478 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.662623 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:21.665603 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:21.665609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:21.665613 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:21.665644 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:21.669737 (+ 4093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:21.670498 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:21.670503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:21.670507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:21.670660 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:21.671161 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:21.671169 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:21.671174 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:21.671599 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:21.671751 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:21.672286 (+ 535us) server_negotiation.cc:299] Negotiation successful
0925 02:57:21.672554 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:57:21.674811 25983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:21.646100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:44133 (local address 127.0.0.1:47238)
0925 02:57:21.657929 (+ 11829us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:21.657949 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:21.658199 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:21.658662 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:21.658667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:21.659304 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:21.659628 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:21.659641 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.660686 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.660689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:21.661675 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:21.661684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.663081 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.663087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:21.663292 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:21.663936 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:21.664140 (+ 204us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:21.665382 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:21.669867 (+ 4485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:21.669872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:21.669878 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:21.670390 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:21.670805 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:21.670812 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:21.670817 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:21.671040 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:21.671821 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:21.671828 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:21.672006 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 02:57:21.673808 (+ 1802us) client_negotiation.cc:240] Negotiation successful
0925 02:57:21.674181 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11687,"thread_start_us":11909,"threads_started":1}
WARNING: no policy specified for oryx/127.18.196.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.196.252@KRBTEST.COM" created.
Sep 25 02:57:21 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305039, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.254@KRBTEST.COM for oryx/127.18.196.253@KRBTEST.COM
I20260925 02:57:21.691903 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:21.670465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47244 (local address 127.18.196.253:44133)
0925 02:57:21.673762 (+ 3297us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:21.673770 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:21.676929 (+ 3159us) server_negotiation.cc:407] Connection header received
0925 02:57:21.677242 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:21.677248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:21.677342 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:21.677435 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:21.678819 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.679742 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.681643 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.682451 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.682582 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:21.686378 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:21.686383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:21.686386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:21.686413 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:21.689203 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:21.689882 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:21.689887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:21.689891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:21.690014 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:21.690414 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:21.690421 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:21.690424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:21.690675 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:21.690830 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:21.691307 (+ 477us) server_negotiation.cc:299] Negotiation successful
0925 02:57:21.691532 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3167,"thread_start_us":6204,"threads_started":1}
I20260925 02:57:21.693045 25985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:21.670130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:44133 (local address 127.0.0.1:47244)
0925 02:57:21.676792 (+ 6662us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:21.676831 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:21.677012 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:21.677529 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:21.677538 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:21.678355 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:21.678638 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:21.678651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.679891 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.679897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:21.681470 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:21.681495 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.682628 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.682635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:21.682874 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:21.683579 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:21.683603 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:21.686189 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:21.689336 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:21.689347 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:21.689355 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:21.689757 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:21.690147 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:21.690152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:21.690156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:21.690305 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:21.690811 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:21.690819 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:21.691021 (+ 202us) client_negotiation.cc:769] Sending connection context
0925 02:57:21.691221 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 02:57:21.691532 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6416,"thread_start_us":5953,"threads_started":1}
I20260925 02:57:21.695050 25431 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.18.196.253:44133: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ab6a987abd184695be36cdde2d06d24e"
member_type: NON_VOTER
last_known_addr {
host: "127.18.196.253"
port: 44133
}
attrs {
promote: true
}
}
dest_uuid: "d736dfd4e08d41148d0bbc56465191ca"
cas_config_opid_index: -1
I20260925 02:57:21.696067 25431 consensus_queue.cc:237] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: NON_VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: true } }
I20260925 02:57:21.703220 25989 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ab6a987abd184695be36cdde2d06d24e (127.18.196.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: NON_VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: true } } }
I20260925 02:57:21.703883 25934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:21.706815 25934 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [term 1 FOLLOWER]: Refusing update from remote peer d736dfd4e08d41148d0bbc56465191ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 02:57:21.707911 25990 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: NON_VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.196.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.252.keytab.
Entry for principal oryx/127.18.196.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.252.keytab.
I20260925 02:57:21.713567 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.252:33819
--webserver_interface=127.18.196.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.252.keytab
--principal=oryx/127.18.196.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:40775,127.18.196.253:44133,127.18.196.252:33819
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:57:21.714927 25989 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.18.196.253:44133
I20260925 02:57:21.717054 25892 master.cc:537] Master@127.18.196.253:44133 shutting down...
I20260925 02:57:21.723683 25990 sys_catalog.cc:455] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: NON_VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: true } } }
I20260925 02:57:21.724134 25990 sys_catalog.cc:458] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:21.744104 25934 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ab6a987abd184695be36cdde2d06d24e (127.18.196.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: NON_VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: true } } }
I20260925 02:57:21.755368 25999 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: NON_VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: true } } }
I20260925 02:57:21.755796 25999 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [sys.catalog]: This master's current role is: LEARNER
I20260925 02:57:21.756198 25965 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d736dfd4e08d41148d0bbc56465191ca. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: NON_VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: true } } }
I20260925 02:57:21.756431 25965 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [sys.catalog]: This master's current role is: LEARNER
I20260925 02:57:21.756757 25996 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: NON_VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: true } } }
I20260925 02:57:21.757092 25996 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [sys.catalog]: This master's current role is: LEARNER
I20260925 02:57:21.773137 25892 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [term 1 LEARNER]: Raft consensus shutting down.
I20260925 02:57:21.773777 25892 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [term 1 LEARNER]: Raft consensus is shut down!
I20260925 02:57:21.773993 25892 tablet_replica.cc:330] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e: stopping tablet replica
I20260925 02:57:21.783551 25892 master.cc:552] Master@127.18.196.253:44133 shutdown complete.
I20260925 02:57:21.783738 25892 master_runner.cc:305] Clearing existing system tablet
I20260925 02:57:21.784286 25892 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:21.784447 25892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:21.787508 25892 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 02:57:21.789786 26001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:21.790140 25892 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:21.790571 25892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 02:57:21.790704 25892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/wal
uuid: "ab6a987abd184695be36cdde2d06d24e"
format_stamp: "Formatted at 2026-09-25 02:57:21 on dist-test-slave-8zvk"
server_key: "7aa926b9a864a17c621becc9bcf34873"
server_key_iv: "597f0104855a5d613bdd81ecacba10ae"
server_key_version: "encryptionkey@0"
I20260925 02:57:21.795435 25892 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:57:21.802376 25892 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 02:57:21.802629 25892 log.cc:1196] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 02:57:21.803198 25892 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e: Deleting consensus metadata
I20260925 02:57:21.803503 25892 master_runner.cc:315] Copying system tablet from 127.18.196.254:40775
I20260925 02:57:21.806423 25892 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e: tablet copy: Beginning tablet copy session from remote peer at address 127.18.196.254:40775
I20260925 02:57:21.823006 25622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:21.807432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43308 (local address 127.18.196.254:40775)
0925 02:57:21.807619 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:21.807626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:21.808143 (+ 517us) server_negotiation.cc:407] Connection header received
0925 02:57:21.808348 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:21.808355 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:21.808458 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:21.808572 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:21.809621 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.810750 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.812702 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.813472 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.813626 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:21.816000 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:21.816009 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:21.816017 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:21.816067 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:21.819591 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:21.820371 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:21.820377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:21.820380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:21.820531 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:21.820946 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:21.820951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:21.820954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:21.821334 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:21.821925 (+ 591us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:21.822362 (+ 437us) server_negotiation.cc:299] Negotiation successful
0925 02:57:21.822600 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:57:21.823441 26007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:21.807378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.0.0.1:43308)
0925 02:57:21.808017 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:21.808036 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:21.808203 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:21.808618 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:21.808624 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:21.809160 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:21.809433 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:21.809446 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.810956 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.810963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:21.812540 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:21.812556 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:21.813656 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:21.813662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:21.813886 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:21.814586 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:21.814609 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:21.815761 (+ 1152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:21.819730 (+ 3969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:21.819735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:21.819740 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:21.820237 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:21.820650 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:21.820655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:21.820659 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:21.820827 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:21.821882 (+ 1055us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:21.821887 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:21.822081 (+ 194us) client_negotiation.cc:769] Sending connection context
0925 02:57:21.822237 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 02:57:21.822554 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":365,"threads_started":1}
I20260925 02:57:21.824357 25451 tablet_copy_service.cc:140] P d736dfd4e08d41148d0bbc56465191ca: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ab6a987abd184695be36cdde2d06d24e ({username='oryx', principal='oryx/127.18.196.253@KRBTEST.COM'} at 127.0.0.1:43308)
I20260925 02:57:21.824652 25451 tablet_copy_service.cc:161] P d736dfd4e08d41148d0bbc56465191ca: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ab6a987abd184695be36cdde2d06d24e at {username='oryx', principal='oryx/127.18.196.253@KRBTEST.COM'} at 127.0.0.1:43308: session id = ab6a987abd184695be36cdde2d06d24e-00000000000000000000000000000000
I20260925 02:57:21.829377 25451 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 02:57:21.832216 25892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 02:57:21.839047 25892 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e: tablet copy: Starting download of 0 data blocks...
I20260925 02:57:21.839399 25892 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e: tablet copy: Starting download of 1 WAL segments...
I20260925 02:57:21.842648 25892 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 02:57:21.847098 25451 tablet_copy_service.cc:342] P d736dfd4e08d41148d0bbc56465191ca: Request end of tablet copy session ab6a987abd184695be36cdde2d06d24e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.18.196.253@KRBTEST.COM'} at 127.0.0.1:43308
I20260925 02:57:21.847325 25451 tablet_copy_service.cc:433] P d736dfd4e08d41148d0bbc56465191ca: ending tablet copy session ab6a987abd184695be36cdde2d06d24e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ab6a987abd184695be36cdde2d06d24e
I20260925 02:57:21.858150 25892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:21.860673 26008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:21.865437 26009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:21.873035 25892 server_base.cc:1034] running on GCE node
W20260925 02:57:21.873427 26011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:21.873939 25892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:21.874622 25892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:21.875844 25892 hybrid_clock.cc:648] HybridClock initialized: now 1790305041875776 us; error 25 us; skew 500 ppm
I20260925 02:57:21.876109 25892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:21 dist-test-slave-8zvk krb5kdc[25386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305041, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:21.878656 25892 init.cc:377] Logged in from keytab as oryx/127.18.196.253@KRBTEST.COM (short username oryx)
I20260925 02:57:21.889539 25892 webserver.cc:460] Webserver started at http://127.18.196.253:46735/ using document root <none> and password file <none>
I20260925 02:57:21.889952 25892 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:21.890062 25892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:21.893548 25892 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260925 02:57:21.895510 26018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:21.895918 25892 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260925 02:57:21.896091 25892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/wal
uuid: "ab6a987abd184695be36cdde2d06d24e"
format_stamp: "Formatted at 2026-09-25 02:57:21 on dist-test-slave-8zvk"
server_key: "7aa926b9a864a17c621becc9bcf34873"
server_key_iv: "597f0104855a5d613bdd81ecacba10ae"
server_key_version: "encryptionkey@0"
I20260925 02:57:21.896369 25892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:21.913714 25892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:21.918344 25892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:21.978946 25892 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.253:44133
I20260925 02:57:21.979012 26069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.253:44133 every 8 connection(s)
I20260925 02:57:21.979979 25892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-1/data/info.pb
I20260925 02:57:21.985924 26070 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:57:21.990823 26070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e: Bootstrap starting.
I20260925 02:57:22.007797 26070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:57:22.008424 26070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e: Bootstrap complete.
I20260925 02:57:22.009645 26070 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: NON_VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: true } }
I20260925 02:57:22.009945 26070 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab6a987abd184695be36cdde2d06d24e, State: Initialized, Role: LEARNER
I20260925 02:57:22.010394 26070 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: NON_VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: true } }
I20260925 02:57:22.011307 26073 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: NON_VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: true } } }
I20260925 02:57:22.011715 26073 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [sys.catalog]: This master's current role is: LEARNER
I20260925 02:57:22.011821 26070 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:22.028147 26084 catalog_manager.cc:1259] Loaded cluster ID: 81e88c27e36e498f87565d50facd8f55
I20260925 02:57:22.028270 26084 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e: loading cluster ID for follower catalog manager: success
I20260925 02:57:22.032815 26084 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e: acquiring CA information for follower catalog manager: success
I20260925 02:57:22.036211 26084 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:57:22.101509 25887 heartbeater.cc:503] Master 127.18.196.254:40775 was elected leader, sending a full tablet report...
I20260925 02:57:22.115610 25989 sys_catalog.cc:455] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: NON_VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: true } } }
I20260925 02:57:22.115964 25989 sys_catalog.cc:458] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:22.131368 25985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:22.115965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:44133 (local address 127.0.0.1:47258)
0925 02:57:22.116360 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:22.116383 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:22.116526 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:22.123919 (+ 7393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:22.123925 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:22.123961 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:22.124157 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:22.124170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.126186 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.126193 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:22.128516 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:22.128543 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.130438 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.130446 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:22.130658 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:22.130737 (+ 79us) client_negotiation.cc:769] Sending connection context
0925 02:57:22.130907 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 02:57:22.131066 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255}
I20260925 02:57:22.131462 26085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:22.117084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47258 (local address 127.18.196.253:44133)
0925 02:57:22.123437 (+ 6353us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:22.123444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:22.123476 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:57:22.123562 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:22.123567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:22.123668 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:22.123822 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:22.124347 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.126015 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.128733 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.130225 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.130452 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:22.130666 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:22.130913 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 02:57:22.131067 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6188,"thread_start_us":5970,"threads_started":1}
I20260925 02:57:22.132853 26073 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d736dfd4e08d41148d0bbc56465191ca. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: NON_VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: true } } }
I20260925 02:57:22.133111 26073 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [sys.catalog]: This master's current role is: LEARNER
I20260925 02:57:22.134078 25991 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca: attempting to promote NON_VOTER ab6a987abd184695be36cdde2d06d24e to VOTER
I20260925 02:57:22.135174 25991 consensus_queue.cc:237] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } }
I20260925 02:57:22.137328 26045 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [term 1 LEARNER]: Refusing update from remote peer d736dfd4e08d41148d0bbc56465191ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 02:57:22.137950 25989 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 02:57:22.141326 25991 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab6a987abd184695be36cdde2d06d24e (127.18.196.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } }
I20260925 02:57:22.146651 25990 sys_catalog.cc:455] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } }
I20260925 02:57:22.146221 26045 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab6a987abd184695be36cdde2d06d24e (127.18.196.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } }
I20260925 02:57:22.146896 25990 sys_catalog.cc:458] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:22.147317 25990 sys_catalog.cc:455] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } }
I20260925 02:57:22.147643 25990 sys_catalog.cc:458] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: This master's current role is: LEADER
W20260925 02:57:22.150482 25993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:22.150981 25993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260925 02:57:22.150982 26073 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } }
W20260925 02:57:22.151212 25993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 02:57:22.151249 26073 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [sys.catalog]: This master's current role is: FOLLOWER
W20260925 02:57:22.158602 25993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:22.158691 25993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:22.158793 25993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:22.158919 25993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:22.158982 25993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:22.169875 25993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:40775,127.18.196.253:44133,127.18.196.252:33819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.252.keytab
--principal=oryx/127.18.196.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.252:33819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.196.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:22.171393 25993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:22.172652 25993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:22.186503 26091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:22.186582 26092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:22.186544 26094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:22.188618 25993 server_base.cc:1034] running on GCE node
I20260925 02:57:22.189436 25993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:22.190891 25993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:22.192116 25993 hybrid_clock.cc:648] HybridClock initialized: now 1790305042192058 us; error 43 us; skew 500 ppm
I20260925 02:57:22.192421 25993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:22 dist-test-slave-8zvk krb5kdc[25386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305042, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:22.200644 25993 init.cc:377] Logged in from keytab as oryx/127.18.196.252@KRBTEST.COM (short username oryx)
I20260925 02:57:22.202595 25993 webserver.cc:460] Webserver started at http://127.18.196.252:33471/ using document root <none> and password file <none>
I20260925 02:57:22.203532 25993 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:22.203711 25993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:22.204080 25993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:22.207775 25993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/data/instance:
uuid: "469b93b3d2084e29807a456e82a08b40"
format_stamp: "Formatted at 2026-09-25 02:57:22 on dist-test-slave-8zvk"
server_key: "414fa2bc4e306dec8130c0726d0f218b"
server_key_iv: "6b6534232a2f95ad0a31ce48d8200943"
server_key_version: "encryptionkey@0"
I20260925 02:57:22.208599 25993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/wal/instance:
uuid: "469b93b3d2084e29807a456e82a08b40"
format_stamp: "Formatted at 2026-09-25 02:57:22 on dist-test-slave-8zvk"
server_key: "414fa2bc4e306dec8130c0726d0f218b"
server_key_iv: "6b6534232a2f95ad0a31ce48d8200943"
server_key_version: "encryptionkey@0"
I20260925 02:57:22.214099 25993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 02:57:22.217829 26101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:22.219223 25993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:57:22.219389 25993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/wal
uuid: "469b93b3d2084e29807a456e82a08b40"
format_stamp: "Formatted at 2026-09-25 02:57:22 on dist-test-slave-8zvk"
server_key: "414fa2bc4e306dec8130c0726d0f218b"
server_key_iv: "6b6534232a2f95ad0a31ce48d8200943"
server_key_version: "encryptionkey@0"
I20260925 02:57:22.219585 25993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:22.231761 25993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:22.235698 25993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:22.235986 25993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:22.259084 25993 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.252:33819
I20260925 02:57:22.259104 26152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.252:33819 every 8 connection(s)
I20260925 02:57:22.260454 25993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/data/info.pb
I20260925 02:57:22.261225 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 25993
I20260925 02:57:22.265179 26153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:22.272572 26153 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:57:22 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.196.252@KRBTEST.COM
Sep 25 02:57:22 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305042, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.252@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM
I20260925 02:57:22.285573 26156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:22.263683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39486 (local address 127.18.196.252:33819)
0925 02:57:22.264405 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:22.264417 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:22.264474 (+ 57us) server_negotiation.cc:407] Connection header received
0925 02:57:22.266319 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:22.266340 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:22.266706 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:22.267260 (+ 554us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:22.268915 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.270439 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.271790 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.272458 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.272675 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:22.277702 (+ 5027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:22.277710 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:22.277720 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:22.277762 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:22.281067 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:22.281767 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:22.281773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:22.281779 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:22.281944 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:22.282531 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:22.282534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:22.282537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:22.283122 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:22.283452 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:22.283902 (+ 450us) server_negotiation.cc:299] Negotiation successful
0925 02:57:22.284390 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":328,"threads_started":1}
I20260925 02:57:22.290947 26157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:22.275203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.0.0.1:43322)
0925 02:57:22.276067 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:22.276113 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:22.276269 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:22.276695 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:22.276705 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:22.277694 (+ 989us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:22.277981 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:22.277990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.279531 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.279537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:22.281040 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:22.281053 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.281872 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.281878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:22.282043 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:22.282624 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:22.282642 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:22.284945 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:22.288755 (+ 3810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:22.288764 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:22.288768 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:22.289201 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:22.289517 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:22.289520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:22.289522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:22.289643 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:22.290099 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:22.290104 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:22.290241 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 02:57:22.290389 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 02:57:22.290609 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"spinlock_wait_cycles":15104,"thread_start_us":424,"threads_started":1}
I20260925 02:57:22.290966 25622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:22.275266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43322 (local address 127.18.196.254:40775)
0925 02:57:22.275482 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:22.275491 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:22.276213 (+ 722us) server_negotiation.cc:407] Connection header received
0925 02:57:22.276409 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:22.276415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:22.276494 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:22.276576 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:22.278152 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.279399 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.281188 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.281667 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.281800 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:22.285158 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:22.285165 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:22.285170 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:22.285238 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:22.288602 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:22.289301 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:22.289305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:22.289307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:22.289416 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:22.289731 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:22.289734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:22.289736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:22.289999 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:22.290112 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:22.290475 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 02:57:22.290646 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:57:22.292315 26153 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:57:22 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305042, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.252@KRBTEST.COM for oryx/127.18.196.253@KRBTEST.COM
I20260925 02:57:22.304600 26157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:22.292925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:44133 (local address 127.0.0.1:47262)
0925 02:57:22.293133 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:22.293153 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:22.293249 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:22.293565 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:22.293569 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:22.293899 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:22.294036 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:22.294044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.295119 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.295128 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:22.296186 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:22.296206 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.296951 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.296967 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:22.297103 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:22.297625 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:22.297641 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:22.299536 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:22.302621 (+ 3085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:22.302625 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:22.302627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:22.303070 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:22.303408 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:22.303411 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:22.303413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:22.303535 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:22.303959 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:22.303962 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:22.304074 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 02:57:22.304195 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 02:57:22.304386 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 02:57:22.304659 26085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:22.293070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47262 (local address 127.18.196.253:44133)
0925 02:57:22.293266 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:22.293272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:22.293297 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:57:22.293357 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:22.293362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:22.293429 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:22.293549 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:22.294187 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.294986 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.296398 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.296816 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.296916 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:22.299719 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:22.299724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:22.299727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:22.299757 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:22.302483 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:22.303180 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:22.303184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:22.303186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:22.303312 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:22.303624 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:22.303627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:22.303629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:22.303866 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:22.303971 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:22.304275 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 02:57:22.304432 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:57:22.305634 26153 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.252" port: 33819 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:57:22 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305042, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.252@KRBTEST.COM for oryx/127.18.196.252@KRBTEST.COM
I20260925 02:57:22.317987 26157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:22.306238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:33819 (local address 127.0.0.1:39498)
0925 02:57:22.306365 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:22.306379 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:22.306483 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:22.306779 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:22.306783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:22.307190 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:22.307425 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:22.307434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.308336 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.308343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:22.309147 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:22.309156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.309939 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.309944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:22.310105 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:22.310733 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:22.310748 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:22.312893 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:22.315676 (+ 2783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:22.315679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:22.315681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:22.316022 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:22.316335 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:22.316338 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:22.316340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:22.316519 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:22.317010 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:22.317013 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:22.317151 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 02:57:22.317341 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 02:57:22.317586 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 02:57:22.318416 26156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:22.306400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39498 (local address 127.18.196.252:33819)
0925 02:57:22.306548 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:22.306553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:22.306568 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:57:22.306620 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:22.306623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:22.306669 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:22.306763 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:22.307557 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.308218 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.309275 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.309787 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.309926 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:22.313051 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:22.313056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:22.313059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:22.313087 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:22.315549 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:22.316127 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:22.316131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:22.316133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:22.316243 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:22.316627 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:22.316632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:22.316639 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:22.316897 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:22.317019 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:22.317429 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 02:57:22.317774 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":46}
I20260925 02:57:22.323194 26153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40: Bootstrap starting.
I20260925 02:57:22.326299 26153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:22.327533 26153 log.cc:824] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:22.330173 26153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40: No bootstrap required, opened a new log
I20260925 02:57:22.337663 26153 raft_consensus.cc:348] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 33819 } }
I20260925 02:57:22.337960 26153 raft_consensus.cc:374] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:22.338078 26153 raft_consensus.cc:729] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469b93b3d2084e29807a456e82a08b40, State: Initialized, Role: FOLLOWER
I20260925 02:57:22.338547 26153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 33819 } }
I20260925 02:57:22.339488 26159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 33819 } } }
I20260925 02:57:22.339872 26159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:57:22.340628 26153 sys_catalog.cc:564] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 02:57:22.350530 26170 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:57:22.350678 26170 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:57:22.354140 25993 master_runner.cc:428] Detected that this master 469b93b3d2084e29807a456e82a08b40 is joining an existing cluster
I20260925 02:57:22.354286 25993 master_runner.cc:432] Initiating AddMaster RPC to add 127.18.196.252:33819
I20260925 02:57:22.377756 25622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:22.359256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43338 (local address 127.18.196.254:40775)
0925 02:57:22.359454 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:22.359461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:22.360370 (+ 909us) server_negotiation.cc:407] Connection header received
0925 02:57:22.360627 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:22.360632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:22.360714 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:22.360782 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:22.362394 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.363557 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.365671 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.366679 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.366784 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:22.369770 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:22.369778 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:22.369783 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:22.369824 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:22.373820 (+ 3996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:22.374808 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:22.374814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:22.374818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:22.375001 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:22.375497 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:22.375503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:22.375508 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:22.375924 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:22.376059 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:22.376792 (+ 733us) server_negotiation.cc:299] Negotiation successful
0925 02:57:22.377343 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:57:22.379142 26176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:22.359060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.0.0.1:43338)
0925 02:57:22.360267 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:22.360290 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:22.360475 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:22.360995 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:22.361001 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:22.361790 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:22.362219 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:22.362233 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.363770 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.363775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:22.365454 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:22.365476 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.367126 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.367133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:22.367318 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:22.368006 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:22.368029 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:22.369415 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:22.374034 (+ 4619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:22.374039 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:22.374043 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:22.374650 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:22.375152 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:22.375157 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:22.375161 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:22.375342 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:22.376161 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:22.376169 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:22.376439 (+ 270us) client_negotiation.cc:769] Sending connection context
0925 02:57:22.377940 (+ 1501us) client_negotiation.cc:240] Negotiation successful
0925 02:57:22.378359 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1046,"thread_start_us":880,"threads_started":1}
I20260925 02:57:22.388816 26156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:22.359458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39510 (local address 127.18.196.252:33819)
0925 02:57:22.359788 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:22.359793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:22.370357 (+ 10564us) server_negotiation.cc:407] Connection header received
0925 02:57:22.370563 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:22.370570 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:22.370665 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:22.370767 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:22.372300 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.373797 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.376766 (+ 2969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.377597 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.377711 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:22.381166 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:22.381172 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:22.381176 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:22.381212 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:22.385081 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:22.385912 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:22.385919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:22.385924 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:22.386088 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:22.386590 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:22.386598 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:22.386604 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:22.387158 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:22.387359 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:22.387978 (+ 619us) server_negotiation.cc:299] Negotiation successful
0925 02:57:22.388256 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42,"spinlock_wait_cycles":34560}
I20260925 02:57:22.391906 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 25492
I20260925 02:57:22.396203 26085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:22.360969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.18.196.253:44133)
0925 02:57:22.361190 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:22.361199 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:22.376611 (+ 15412us) server_negotiation.cc:407] Connection header received
0925 02:57:22.376753 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:22.376758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:22.376846 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:22.376932 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:22.381173 (+ 4241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.382350 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.384781 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.385409 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.385539 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:22.387990 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:22.387997 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:22.388002 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:22.388041 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:22.392255 (+ 4214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:22.393165 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:22.393171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:22.393175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:22.393326 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:22.393800 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:22.393805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:22.393810 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:22.394151 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:22.394304 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:22.395177 (+ 873us) server_negotiation.cc:299] Negotiation successful
0925 02:57:22.395464 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 02:57:22.397256 26177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:22.360441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:44133 (local address 127.0.0.1:47264)
0925 02:57:22.376485 (+ 16044us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:22.376505 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:22.376631 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:22.377118 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:22.377123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:22.380110 (+ 2987us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:22.380965 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:22.380978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.382563 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.382570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:22.384590 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:22.384606 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.385788 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.385794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:22.386001 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:22.386606 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:22.386629 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:22.387758 (+ 1129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:22.392435 (+ 4677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:22.392439 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:22.392443 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:22.393016 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:22.393487 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:22.393491 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:22.393495 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:22.393674 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:22.394416 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:22.394421 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:22.394623 (+ 202us) client_negotiation.cc:769] Sending connection context
0925 02:57:22.396410 (+ 1787us) client_negotiation.cc:240] Negotiation successful
0925 02:57:22.396797 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15900,"thread_start_us":10108,"threads_started":1}
Sep 25 02:57:22 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305039, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.254@KRBTEST.COM for oryx/127.18.196.252@KRBTEST.COM
I20260925 02:57:22.401152 26178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:22.360159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:33819 (local address 127.0.0.1:39510)
0925 02:57:22.370227 (+ 10068us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:22.370257 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:22.370415 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:22.370803 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:22.370810 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:22.371700 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:22.372108 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:22.372123 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.374101 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.374109 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:22.376628 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:22.376643 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.378149 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.378158 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:22.378343 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:22.379448 (+ 1105us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:22.379472 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:22.381001 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:22.385213 (+ 4212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:22.385218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:22.385221 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:22.385787 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:22.386241 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:22.386246 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:22.386249 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:22.386431 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:22.387436 (+ 1005us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:22.387444 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:22.387679 (+ 235us) client_negotiation.cc:769] Sending connection context
0925 02:57:22.400088 (+ 12409us) client_negotiation.cc:240] Negotiation successful
0925 02:57:22.400503 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9890,"thread_start_us":16001,"threads_started":1}
I20260925 02:57:22.408079 26156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:22.386154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39524 (local address 127.18.196.252:33819)
0925 02:57:22.389097 (+ 2943us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:22.389102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:22.389117 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:57:22.389223 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:22.389231 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:22.389392 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:22.390514 (+ 1122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:22.390690 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.392121 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.393968 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.394883 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.395741 (+ 858us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:22.400139 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:22.400145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:22.400150 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:22.400188 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:22.404778 (+ 4590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:22.405737 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:22.405742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:22.405746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:22.405885 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:22.406352 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:22.406357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:22.406360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:22.406725 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:22.406901 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:22.407435 (+ 534us) server_negotiation.cc:299] Negotiation successful
0925 02:57:22.407653 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2859,"threads_started":1}
I20260925 02:57:22.408236 25985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:22.385292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:33819 (local address 127.0.0.1:39524)
0925 02:57:22.385704 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:22.385724 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:22.385829 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:22.389627 (+ 3798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:22.389635 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:22.390118 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:22.390355 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:22.390367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.392302 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.392307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:22.393730 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:22.393748 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.395105 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.395114 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:22.395361 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:22.396028 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:22.396057 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:22.399921 (+ 3864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:22.404960 (+ 5039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:22.404968 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:22.404974 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:22.405596 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:22.406029 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:22.406037 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:22.406043 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:22.406229 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:22.406901 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:22.406909 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:22.407133 (+ 224us) client_negotiation.cc:769] Sending connection context
0925 02:57:22.407311 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 02:57:22.407710 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283}
I20260925 02:57:22.412557 25431 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.18.196.252:33819: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "469b93b3d2084e29807a456e82a08b40"
member_type: NON_VOTER
last_known_addr {
host: "127.18.196.252"
port: 33819
}
attrs {
promote: true
}
}
dest_uuid: "d736dfd4e08d41148d0bbc56465191ca"
cas_config_opid_index: 6
I20260925 02:57:22.413458 25431 consensus_queue.cc:237] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: NON_VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: true } }
I20260925 02:57:22.415966 26045 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [term 1 FOLLOWER]: Refusing update from remote peer d736dfd4e08d41148d0bbc56465191ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 02:57:22.416285 26128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:22.420279 26128 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [term 1 FOLLOWER]: Refusing update from remote peer d736dfd4e08d41148d0bbc56465191ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 02:57:22.421099 25991 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: NON_VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:57:22.422683 25991 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
Sep 25 02:57:22 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.196.193@KRBTEST.COM
I20260925 02:57:22.425500 25990 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 469b93b3d2084e29807a456e82a08b40 (127.18.196.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: NON_VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: true } } }
I20260925 02:57:22.429948 25990 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.18.196.252:33819
I20260925 02:57:22.430270 26045 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 469b93b3d2084e29807a456e82a08b40 (127.18.196.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: NON_VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: true } } }
I20260925 02:57:22.430737 25993 master.cc:537] Master@127.18.196.252:33819 shutting down...
I20260925 02:57:22.432667 25991 sys_catalog.cc:455] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: NON_VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: true } } }
I20260925 02:57:22.432914 25991 sys_catalog.cc:458] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:22.433372 26180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:22.393817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46298 (local address 127.18.196.193:38247)
0925 02:57:22.411704 (+ 17887us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:22.411715 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:22.411750 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:57:22.411823 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:22.411831 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:22.412153 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:22.412325 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:22.414149 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.415830 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.418021 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.419191 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.419319 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:22.425037 (+ 5718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:22.425051 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:22.425068 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:22.425110 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:22.429011 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:22.429872 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:22.429881 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:22.429884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:22.430120 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:22.430634 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:22.430639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:22.430642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:22.431081 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:22.432477 (+ 1396us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:22.432671 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 02:57:22.432972 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17614,"thread_start_us":17679,"threads_started":1}
I20260925 02:57:22.433295 25989 sys_catalog.cc:455] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: NON_VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: true } } }
I20260925 02:57:22.433531 25989 sys_catalog.cc:458] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:22.435194 25615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:22.439157 26073 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: NON_VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: true } } }
I20260925 02:57:22.439441 26073 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:57:22.446452 26128 raft_consensus.cc:858] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 33819 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: NON_VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } }
W20260925 02:57:22.446811 26128 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 33819 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: NON_VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } }
I20260925 02:57:22.479168 26128 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ab6a987abd184695be36cdde2d06d24e (127.18.196.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: NON_VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: true } } }
I20260925 02:57:22.483866 26187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: NON_VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: true } } }
I20260925 02:57:22.484318 26187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 02:57:22.484371 26184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: NON_VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: true } } }
I20260925 02:57:22.484750 26184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 02:57:22.484722 26159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d736dfd4e08d41148d0bbc56465191ca. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: NON_VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: true } } }
I20260925 02:57:22.484929 26159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 02:57:22.486565 25990 sys_catalog.cc:455] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: NON_VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: true } } }
I20260925 02:57:22.486864 25990 sys_catalog.cc:458] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: This master's current role is: LEADER
W20260925 02:57:22.487735 25417 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260925 02:57:22.487952 25993 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260925 02:57:22.488356 25993 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260925 02:57:22.488523 25993 tablet_replica.cc:330] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40: stopping tablet replica
W20260925 02:57:22.495151 25417 consensus_peers.cc:487] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca -> Peer 469b93b3d2084e29807a456e82a08b40 (127.18.196.252:33819): Couldn't send request to peer 469b93b3d2084e29807a456e82a08b40. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260925 02:57:22.496784 25993 master.cc:552] Master@127.18.196.252:33819 shutdown complete.
I20260925 02:57:22.496869 25993 master_runner.cc:305] Clearing existing system tablet
I20260925 02:57:22.497309 25993 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:22.497421 25993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:22.500663 25993 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 02:57:22.502424 26190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:22.502933 25993 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:22.503181 25993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260925 02:57:22.503278 25993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/wal
uuid: "469b93b3d2084e29807a456e82a08b40"
format_stamp: "Formatted at 2026-09-25 02:57:22 on dist-test-slave-8zvk"
server_key: "414fa2bc4e306dec8130c0726d0f218b"
server_key_iv: "6b6534232a2f95ad0a31ce48d8200943"
server_key_version: "encryptionkey@0"
I20260925 02:57:22.507874 25993 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:57:22.514806 25993 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 02:57:22.514967 25993 log.cc:1196] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 02:57:22.515501 25993 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40: Deleting consensus metadata
I20260925 02:57:22.515771 25993 master_runner.cc:315] Copying system tablet from 127.18.196.254:40775
I20260925 02:57:22.519086 25993 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40: tablet copy: Beginning tablet copy session from remote peer at address 127.18.196.254:40775
I20260925 02:57:22.535933 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:22.519848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.0.0.1:43346)
0925 02:57:22.520535 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:22.520560 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:22.520703 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:22.521125 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:22.521130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:22.521647 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:22.522101 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:22.522121 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.523714 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.523720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:22.525282 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:22.525296 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.526666 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.526673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:22.526884 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:22.527612 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:22.527634 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:22.528711 (+ 1077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:22.532778 (+ 4067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:22.532785 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:22.532790 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:22.533324 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:22.533843 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:22.533848 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:22.533852 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:22.534024 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:22.534756 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:22.534761 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:22.535002 (+ 241us) client_negotiation.cc:769] Sending connection context
0925 02:57:22.535160 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 02:57:22.535487 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":408,"threads_started":1}
I20260925 02:57:22.536972 25622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:22.519922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43346 (local address 127.18.196.254:40775)
0925 02:57:22.520142 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:22.520148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:22.520662 (+ 514us) server_negotiation.cc:407] Connection header received
0925 02:57:22.520835 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:22.520841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:22.520939 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:22.521063 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:22.522282 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.523506 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.525811 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.526485 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.526623 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:22.528912 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:22.528918 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:22.528924 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:22.528970 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:22.532639 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:22.533459 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:22.533464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:22.533467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:22.533711 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:22.534177 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:22.534182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:22.534186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:22.534544 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:22.534692 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:22.536340 (+ 1648us) server_negotiation.cc:299] Negotiation successful
0925 02:57:22.536603 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 02:57:22.537468 25451 tablet_copy_service.cc:140] P d736dfd4e08d41148d0bbc56465191ca: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 469b93b3d2084e29807a456e82a08b40 ({username='oryx', principal='oryx/127.18.196.252@KRBTEST.COM'} at 127.0.0.1:43346)
I20260925 02:57:22.537638 25451 tablet_copy_service.cc:161] P d736dfd4e08d41148d0bbc56465191ca: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 469b93b3d2084e29807a456e82a08b40 at {username='oryx', principal='oryx/127.18.196.252@KRBTEST.COM'} at 127.0.0.1:43346: session id = 469b93b3d2084e29807a456e82a08b40-00000000000000000000000000000000
I20260925 02:57:22.541941 25451 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 02:57:22.543846 25993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 02:57:22.551517 25993 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40: tablet copy: Starting download of 0 data blocks...
I20260925 02:57:22.551857 25993 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40: tablet copy: Starting download of 1 WAL segments...
I20260925 02:57:22.552387 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 25492
I20260925 02:57:22.554246 25993 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 02:57:22.558247 25451 tablet_copy_service.cc:342] P d736dfd4e08d41148d0bbc56465191ca: Request end of tablet copy session 469b93b3d2084e29807a456e82a08b40-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.18.196.252@KRBTEST.COM'} at 127.0.0.1:43346
I20260925 02:57:22.558362 25451 tablet_copy_service.cc:433] P d736dfd4e08d41148d0bbc56465191ca: ending tablet copy session 469b93b3d2084e29807a456e82a08b40-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 469b93b3d2084e29807a456e82a08b40
I20260925 02:57:22.571375 25993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:22.574131 26197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:22.575340 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:38247
--local_ip_for_outbound_sockets=127.18.196.193
--tserver_master_addrs=127.18.196.254:40775
--webserver_port=42395
--webserver_interface=127.18.196.193
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.193.keytab
--principal=oryx/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.196.254:40775,127.18.196.253:44133
--tserver_master_addrs=127.18.196.254:40775,127.18.196.253:44133,127.18.196.252:33819 with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:22.576284 26198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:22.583550 25993 server_base.cc:1034] running on GCE node
W20260925 02:57:22.587146 26202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:22.587641 25993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:22.588413 25993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:22.589622 25993 hybrid_clock.cc:648] HybridClock initialized: now 1790305042589575 us; error 31 us; skew 500 ppm
I20260925 02:57:22.589908 25993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:22 dist-test-slave-8zvk krb5kdc[25386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305042, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:22.592648 25993 init.cc:377] Logged in from keytab as oryx/127.18.196.252@KRBTEST.COM (short username oryx)
I20260925 02:57:22.594121 25993 webserver.cc:460] Webserver started at http://127.18.196.252:40243/ using document root <none> and password file <none>
I20260925 02:57:22.594518 25993 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:22.594630 25993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:22.607514 25993 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.000s sys 0.012s
I20260925 02:57:22.609449 26208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:22.609884 25993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260925 02:57:22.610008 25993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/wal
uuid: "469b93b3d2084e29807a456e82a08b40"
format_stamp: "Formatted at 2026-09-25 02:57:22 on dist-test-slave-8zvk"
server_key: "414fa2bc4e306dec8130c0726d0f218b"
server_key_iv: "6b6534232a2f95ad0a31ce48d8200943"
server_key_version: "encryptionkey@0"
I20260925 02:57:22.610183 25993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:22.639550 25993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:22.643564 25993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:22.662949 25993 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.252:33819
I20260925 02:57:22.663102 26259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.252:33819 every 8 connection(s)
I20260925 02:57:22.663997 25993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/master-2/data/info.pb
I20260925 02:57:22.669788 26260 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:57:22.672312 26260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40: Bootstrap starting.
I20260925 02:57:22.683893 26260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:57:22.684365 26260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40: Bootstrap complete.
I20260925 02:57:22.685271 26260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: NON_VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: true } }
I20260925 02:57:22.685527 26260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 469b93b3d2084e29807a456e82a08b40, State: Initialized, Role: LEARNER
I20260925 02:57:22.685812 26260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: NON_VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: true } }
I20260925 02:57:22.686633 26263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: NON_VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: true } } }
I20260925 02:57:22.686952 26260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:22.687179 26263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:57:22.698640 26274 catalog_manager.cc:1259] Loaded cluster ID: 81e88c27e36e498f87565d50facd8f55
I20260925 02:57:22.698767 26274 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40: loading cluster ID for follower catalog manager: success
I20260925 02:57:22.702145 26274 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40: acquiring CA information for follower catalog manager: success
I20260925 02:57:22.704542 26274 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 02:57:22.931001 26200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:22.931227 26200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:22.931404 26200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:22.939402 26200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:22.939522 26200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:22.939702 26200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 02:57:22.949383 26200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.193.keytab
--principal=oryx/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:38247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=42395
--tserver_master_addrs=127.18.196.254:40775,127.18.196.253:44133,127.18.196.252:33819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:22.950528 26200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:22.951989 26200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:22.964566 26282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:22.964619 26279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:22.964643 26280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:22.965541 26200 server_base.cc:1034] running on GCE node
I20260925 02:57:22.966149 26200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:22.967059 26200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:22.968281 26200 hybrid_clock.cc:648] HybridClock initialized: now 1790305042968246 us; error 44 us; skew 500 ppm
I20260925 02:57:22.968585 26200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:22 dist-test-slave-8zvk krb5kdc[25386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305042, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:22.971968 26200 init.cc:377] Logged in from keytab as oryx/127.18.196.193@KRBTEST.COM (short username oryx)
I20260925 02:57:22.973456 26200 webserver.cc:460] Webserver started at http://127.18.196.193:42395/ using document root <none> and password file <none>
I20260925 02:57:22.974117 26200 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:22.974277 26200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:22.979519 25991 sys_catalog.cc:455] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: NON_VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: true } } }
I20260925 02:57:22.979836 25991 sys_catalog.cc:458] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:22.980057 26200 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 02:57:22.990398 26291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:22.992508 26200 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.006s
I20260925 02:57:22.992671 26200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "c2fa8d55ea204f48a127e243690cb618"
format_stamp: "Formatted at 2026-09-25 02:57:20 on dist-test-slave-8zvk"
server_key: "97f72b910581987a31feebcc5be1c759"
server_key_iv: "63dc6cba9ef20b973a81e89196e15add"
server_key_version: "encryptionkey@0"
I20260925 02:57:22.993752 26200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:22.999514 26290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:22.980358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39540 (local address 127.18.196.252:33819)
0925 02:57:22.989730 (+ 9372us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:22.989736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:22.992083 (+ 2347us) server_negotiation.cc:407] Connection header received
0925 02:57:22.992292 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:22.992298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:22.992390 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:22.992519 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:22.992957 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.994545 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.997314 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.998489 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.998651 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:22.998762 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:22.999143 (+ 381us) server_negotiation.cc:299] Negotiation successful
0925 02:57:22.999240 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9213,"thread_start_us":9075,"threads_started":1}
I20260925 02:57:22.999749 26289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:22.979840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:33819 (local address 127.0.0.1:39540)
0925 02:57:22.991962 (+ 12122us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:22.991982 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:22.992135 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:22.992561 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:22.992568 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:22.992597 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:22.992829 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:22.992843 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.994721 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.994728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:22.997167 (+ 2439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:22.997183 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:22.998667 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:22.998672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:22.998899 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:22.998953 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 02:57:22.999126 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 02:57:22.999259 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12004,"thread_start_us":12401,"threads_started":1}
I20260925 02:57:23.000715 26235 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260925 02:57:23.001159 26263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d736dfd4e08d41148d0bbc56465191ca. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: NON_VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: true } } }
I20260925 02:57:23.001503 26263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:57:23.001890 25991 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca: attempting to promote NON_VOTER 469b93b3d2084e29807a456e82a08b40 to VOTER
I20260925 02:57:23.002859 25991 consensus_queue.cc:237] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: false } }
I20260925 02:57:23.004746 26045 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [term 1 FOLLOWER]: Refusing update from remote peer d736dfd4e08d41148d0bbc56465191ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 02:57:23.004841 26235 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [term 1 LEARNER]: Refusing update from remote peer d736dfd4e08d41148d0bbc56465191ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 02:57:23.005342 25989 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 02:57:23.005614 25990 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 02:57:23.008149 25989 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 469b93b3d2084e29807a456e82a08b40 (127.18.196.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: false } } }
I20260925 02:57:23.011739 26200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:23.011463 26045 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 469b93b3d2084e29807a456e82a08b40 (127.18.196.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: false } } }
I20260925 02:57:23.011758 26235 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 469b93b3d2084e29807a456e82a08b40 (127.18.196.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: false } } }
I20260925 02:57:23.013307 25990 sys_catalog.cc:455] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: false } } }
I20260925 02:57:23.013666 25990 sys_catalog.cc:458] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:23.014050 25990 sys_catalog.cc:455] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: false } } }
I20260925 02:57:23.014315 25990 sys_catalog.cc:458] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:23.014608 25990 sys_catalog.cc:455] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: false } } }
I20260925 02:57:23.014827 25990 sys_catalog.cc:458] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:23.016156 26297 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: false } } }
I20260925 02:57:23.016434 26297 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:57:23.016470 26263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d736dfd4e08d41148d0bbc56465191ca" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: false } } }
I20260925 02:57:23.016743 26263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:57:23.017558 26200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:23.017836 26200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:23.018806 26200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:23.020685 26200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:23.020793 26200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:23.020922 26200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:23.021021 26200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:23.058512 26200 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:38247
I20260925 02:57:23.058600 26405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:38247 every 8 connection(s)
I20260925 02:57:23.060123 26200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 02:57:23.068336 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 26200
I20260925 02:57:23.068719 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 25626
Sep 25 02:57:23 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305042, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.253@KRBTEST.COM
Sep 25 02:57:23 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305042, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM
Sep 25 02:57:23 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305042, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.252@KRBTEST.COM
Sep 25 02:57:23 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.196.194@KRBTEST.COM
I20260925 02:57:23.101416 26290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:23.063086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:41107 (local address 127.18.196.252:33819)
0925 02:57:23.063327 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:23.063333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:23.073906 (+ 10573us) server_negotiation.cc:407] Connection header received
0925 02:57:23.077352 (+ 3446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:23.077361 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:23.077467 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:23.077537 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:23.080517 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.081313 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.084452 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.085187 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.085287 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:23.093048 (+ 7761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:23.093053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:23.093056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:23.093113 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:23.097011 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:23.098075 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:23.098080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:23.098085 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:23.098234 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:23.098835 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:23.098839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:23.098862 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:23.099264 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:23.099583 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:23.100699 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0925 02:57:23.100934 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:57:23.101754 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:23.063051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:33365 (local address 127.18.196.254:40775)
0925 02:57:23.063772 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:23.063778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:23.066069 (+ 2291us) server_negotiation.cc:407] Connection header received
0925 02:57:23.077469 (+ 11400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:23.077475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:23.077552 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:23.077626 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:23.083254 (+ 5628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.084422 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.086466 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.087214 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.087310 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:23.092618 (+ 5308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:23.092626 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:23.092633 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:23.092680 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:23.097328 (+ 4648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:23.098355 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:23.098360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:23.098364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:23.098623 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:23.099356 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:23.099361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:23.099376 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:23.099936 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:23.100095 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:23.100811 (+ 716us) server_negotiation.cc:299] Negotiation successful
0925 02:57:23.101030 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":436,"threads_started":1}
I20260925 02:57:23.102874 26413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:23.063747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:33819 (local address 127.18.196.193:41107)
0925 02:57:23.073786 (+ 10039us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:23.073808 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:23.076965 (+ 3157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:23.078062 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:23.078073 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:23.079285 (+ 1212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:23.080282 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:23.080320 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.081780 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.081787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:23.084289 (+ 2502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:23.084303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.085683 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.085694 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:23.086047 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:23.087041 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:23.087071 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:23.092782 (+ 5711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:23.097154 (+ 4372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:23.097158 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:23.097161 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:23.097942 (+ 781us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:23.098373 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:23.098389 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:23.098392 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:23.098721 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:23.099683 (+ 962us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:23.099691 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:23.100207 (+ 516us) client_negotiation.cc:769] Sending connection context
0925 02:57:23.101688 (+ 1481us) client_negotiation.cc:240] Negotiation successful
0925 02:57:23.101963 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9883,"mutex_wait_us":152,"spinlock_wait_cycles":43520,"thread_start_us":16618,"threads_started":1}
I20260925 02:57:23.102819 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:23.064939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:44133 (local address 127.18.196.193:40623)
0925 02:57:23.082050 (+ 17111us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:23.082067 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:23.082186 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:23.082593 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:23.082597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:23.083319 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:23.083664 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:23.083676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.085436 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.085442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:23.087193 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:23.087205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.088103 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.088109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:23.088317 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:23.088868 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:23.088896 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:23.091679 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:23.096924 (+ 5245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:23.096935 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:23.096952 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:23.097942 (+ 990us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:23.098373 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:23.098378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:23.098381 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:23.098669 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:23.099696 (+ 1027us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:23.099700 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:23.100207 (+ 507us) client_negotiation.cc:769] Sending connection context
0925 02:57:23.100556 (+ 349us) client_negotiation.cc:240] Negotiation successful
0925 02:57:23.101077 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16979,"mutex_wait_us":16,"spinlock_wait_cycles":12288,"thread_start_us":8512,"threads_started":1}
I20260925 02:57:23.103830 26414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:23.065030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:40623 (local address 127.18.196.253:44133)
0925 02:57:23.076490 (+ 11460us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:23.076496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:23.082288 (+ 5792us) server_negotiation.cc:407] Connection header received
0925 02:57:23.082354 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:23.082360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:23.082435 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:23.083813 (+ 1378us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:23.084015 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.085264 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.087319 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.087914 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.088042 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:23.091970 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:23.091978 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:23.091986 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:23.092028 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:23.096739 (+ 4711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:23.098115 (+ 1376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:23.098121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:23.098126 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:23.098253 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:23.098831 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:23.098835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:23.098838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:23.099547 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:23.102966 (+ 3419us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:23.103138 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 02:57:23.103336 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11261,"thread_start_us":11593,"threads_started":1}
I20260925 02:57:23.104264 26416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:23.069909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50682 (local address 127.18.196.194:36001)
0925 02:57:23.078451 (+ 8542us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:23.078461 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:23.078518 (+ 57us) server_negotiation.cc:407] Connection header received
0925 02:57:23.078589 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:23.078597 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:23.078877 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:23.079025 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:23.081265 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.082935 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.084989 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.085857 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.085976 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:23.094998 (+ 9022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:23.095015 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:23.095034 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:23.095085 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:23.099734 (+ 4649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:23.100794 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:23.100802 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:23.100805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:23.100959 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:23.101415 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:23.101420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:23.101423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:23.101924 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:23.102328 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:23.103593 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0925 02:57:23.103882 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8156,"thread_start_us":8014,"threads_started":1}
I20260925 02:57:23.104844 26406 heartbeater.cc:347] Connected to a master server at 127.18.196.252:33819
I20260925 02:57:23.105144 26406 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:23.105578 26408 heartbeater.cc:347] Connected to a master server at 127.18.196.253:44133
I20260925 02:57:23.105659 25749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:23.105703 26408 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:23.106565 26406 heartbeater.cc:511] Master 127.18.196.252:33819 requested a full tablet report, sending...
I20260925 02:57:23.106593 26408 heartbeater.cc:511] Master 127.18.196.253:44133 requested a full tablet report, sending...
I20260925 02:57:23.107723 26411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:23.063733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.18.196.193:33365)
0925 02:57:23.064648 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:23.064709 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:23.076965 (+ 12256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:23.078062 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:23.078073 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:23.082285 (+ 4212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:23.082770 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:23.082783 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.084649 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.084654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:23.086288 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:23.086302 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.087637 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.087643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:23.087828 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:23.088428 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:23.088455 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:23.092327 (+ 3872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:23.097533 (+ 5206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:23.097541 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:23.097548 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:23.098221 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:23.098793 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:23.098801 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:23.098806 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:23.099173 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:23.100217 (+ 1044us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:23.100222 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:23.100477 (+ 255us) client_negotiation.cc:769] Sending connection context
0925 02:57:23.102224 (+ 1747us) client_negotiation.cc:240] Negotiation successful
0925 02:57:23.102558 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":27,"spinlock_wait_cycles":768,"thread_start_us":351,"threads_started":1}
I20260925 02:57:23.108911 26035 ts_manager.cc:194] Registered new tserver with Master: c2fa8d55ea204f48a127e243690cb618 (127.18.196.193:38247)
I20260925 02:57:23.109323 26225 ts_manager.cc:194] Registered new tserver with Master: c2fa8d55ea204f48a127e243690cb618 (127.18.196.193:38247)
I20260925 02:57:23.109509 26407 heartbeater.cc:347] Connected to a master server at 127.18.196.254:40775
I20260925 02:57:23.109642 26407 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:23.110131 26407 heartbeater.cc:511] Master 127.18.196.254:40775 requested a full tablet report, sending...
I20260925 02:57:23.111212 25431 ts_manager.cc:194] Re-registered known tserver with Master: c2fa8d55ea204f48a127e243690cb618 (127.18.196.193:38247)
I20260925 02:57:23.112214 25431 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.196.193@KRBTEST.COM'} at 127.18.196.193:33365
I20260925 02:57:23.192417 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 25626
I20260925 02:57:23.209815 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:36001
--local_ip_for_outbound_sockets=127.18.196.194
--tserver_master_addrs=127.18.196.254:40775
--webserver_port=46411
--webserver_interface=127.18.196.194
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.194.keytab
--principal=oryx/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.196.254:40775,127.18.196.253:44133
--tserver_master_addrs=127.18.196.254:40775,127.18.196.253:44133,127.18.196.252:33819 with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:23.534627 26418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:23.534998 26418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:23.535153 26418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:23.546715 26418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:23.546864 26418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:23.547044 26418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 02:57:23.558094 26418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.194.keytab
--principal=oryx/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:36001
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=46411
--tserver_master_addrs=127.18.196.254:40775,127.18.196.253:44133,127.18.196.252:33819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:23.559428 26418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:23.560664 26418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:23.574398 26423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:23.574407 26424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:23.574496 26426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:23.575219 26418 server_base.cc:1034] running on GCE node
I20260925 02:57:23.575855 26418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:23.576890 26418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:23.578123 26418 hybrid_clock.cc:648] HybridClock initialized: now 1790305043578082 us; error 40 us; skew 500 ppm
I20260925 02:57:23.578435 26418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:23 dist-test-slave-8zvk krb5kdc[25386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305043, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:23.581724 26418 init.cc:377] Logged in from keytab as oryx/127.18.196.194@KRBTEST.COM (short username oryx)
I20260925 02:57:23.583233 26418 webserver.cc:460] Webserver started at http://127.18.196.194:46411/ using document root <none> and password file <none>
I20260925 02:57:23.583886 26418 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:23.584038 26418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:23.589450 26418 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:57:23.592128 26433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:23.593418 26418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:57:23.593587 26418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "bb0d89109f23435db76236a5f64b25d5"
format_stamp: "Formatted at 2026-09-25 02:57:20 on dist-test-slave-8zvk"
server_key: "19cc7f7ee7df85391fd716127804dfdc"
server_key_iv: "970a24ac0927882dc0bad979a20dcfac"
server_key_version: "encryptionkey@0"
I20260925 02:57:23.594532 26418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:23.635236 26418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:23.639570 26418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:23.639868 26418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:23.640846 26418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:23.642156 26418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:23.642272 26418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:23.642403 26418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:23.642504 26418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:23.679996 26418 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:36001
I20260925 02:57:23.680083 26545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:36001 every 8 connection(s)
I20260925 02:57:23.681459 26418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 02:57:23.683580 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 26418
I20260925 02:57:23.683877 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 25759
Sep 25 02:57:23 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305043, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM
Sep 25 02:57:23 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305043, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.252@KRBTEST.COM
Sep 25 02:57:23 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.196.195@KRBTEST.COM
I20260925 02:57:23.736691 26552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:23.684639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:42205 (local address 127.18.196.252:33819)
0925 02:57:23.712770 (+ 28131us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:23.712777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:23.712804 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:57:23.714250 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:23.714259 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:23.714390 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:23.714514 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:23.717490 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.719002 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.721963 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.722867 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.723004 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:23.728915 (+ 5911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:23.728938 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:23.728945 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:23.729001 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:23.732947 (+ 3946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:23.733736 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:23.733741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:23.733744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:23.733901 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:23.734460 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:23.734464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:23.734467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:23.734869 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:23.735004 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:23.736072 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0925 02:57:23.736300 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27981,"thread_start_us":27894,"threads_started":1}
I20260925 02:57:23.737577 26553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:23.685158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60552 (local address 127.18.196.195:43605)
0925 02:57:23.713281 (+ 28123us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:23.713293 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:23.714051 (+ 758us) server_negotiation.cc:407] Connection header received
0925 02:57:23.714132 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:23.714140 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:23.714400 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:23.714547 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:23.716981 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.718751 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.720831 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.721709 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.721804 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:23.729375 (+ 7571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:23.729408 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:23.729429 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:23.729477 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:23.733815 (+ 4338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:23.734804 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:23.734814 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:23.734820 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:23.735047 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:23.735564 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:23.735571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:23.735574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:23.736023 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:23.736292 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:23.736908 (+ 616us) server_negotiation.cc:299] Negotiation successful
0925 02:57:23.737177 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27940,"thread_start_us":28120,"threads_started":1}
I20260925 02:57:23.738586 26550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:23.684208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:45233 (local address 127.18.196.254:40775)
0925 02:57:23.711390 (+ 27182us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:23.711397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:23.713259 (+ 1862us) server_negotiation.cc:407] Connection header received
0925 02:57:23.714231 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:23.714237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:23.714320 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:23.714440 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:23.717494 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.718748 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.721732 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.722332 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.722450 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:23.728876 (+ 6426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:23.728886 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:23.728894 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:23.728940 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:23.734127 (+ 5187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:23.735304 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:23.735309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:23.735312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:23.735502 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:23.735965 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:23.735971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:23.735974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:23.736327 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:23.736473 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:23.737991 (+ 1518us) server_negotiation.cc:299] Negotiation successful
0925 02:57:23.738265 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27019,"thread_start_us":26920,"threads_started":1}
I20260925 02:57:23.739477 25882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:23.740042 26555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:23.685095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.18.196.194:45233)
0925 02:57:23.713065 (+ 27970us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:23.713091 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:23.713798 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:23.714913 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:23.714928 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:23.716098 (+ 1170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:23.717232 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:23.717309 (+ 77us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.719072 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.719076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:23.721559 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:23.721575 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.722819 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.722829 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:23.723196 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:23.724108 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:23.724148 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:23.728579 (+ 4431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:23.734283 (+ 5704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:23.734287 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:23.734291 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:23.735165 (+ 874us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:23.735637 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:23.735649 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:23.735652 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:23.735824 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:23.736539 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:23.736545 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:23.736693 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 02:57:23.736868 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 02:57:23.737388 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27791,"mutex_wait_us":619,"thread_start_us":27014,"threads_started":1}
I20260925 02:57:23.740056 26554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:23.685094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:33819 (local address 127.18.196.194:42205)
0925 02:57:23.710241 (+ 25147us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:23.710296 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:23.713792 (+ 3496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:23.714916 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:23.714936 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:23.716099 (+ 1163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:23.717233 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:23.717265 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.719226 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.719234 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:23.721737 (+ 2503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:23.721758 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.723409 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.723418 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:23.723681 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:23.724417 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:23.724450 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:23.728581 (+ 4131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:23.733148 (+ 4567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:23.733158 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:23.733175 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:23.733603 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:23.734055 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:23.734059 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:23.734063 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:23.734324 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:23.735122 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:23.735134 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:23.735615 (+ 481us) client_negotiation.cc:769] Sending connection context
0925 02:57:23.737136 (+ 1521us) client_negotiation.cc:240] Negotiation successful
0925 02:57:23.738271 (+ 1135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24712,"mutex_wait_us":137,"spinlock_wait_cycles":61440,"thread_start_us":24449,"threads_started":1}
I20260925 02:57:23.741693 26547 heartbeater.cc:347] Connected to a master server at 127.18.196.254:40775
I20260925 02:57:23.742012 26547 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:23.742807 26546 heartbeater.cc:347] Connected to a master server at 127.18.196.252:33819
I20260925 02:57:23.743021 26547 heartbeater.cc:511] Master 127.18.196.254:40775 requested a full tablet report, sending...
I20260925 02:57:23.743033 26546 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:23.744349 26546 heartbeater.cc:511] Master 127.18.196.252:33819 requested a full tablet report, sending...
I20260925 02:57:23.745445 25431 ts_manager.cc:194] Re-registered known tserver with Master: bb0d89109f23435db76236a5f64b25d5 (127.18.196.194:36001)
I20260925 02:57:23.745618 26225 ts_manager.cc:194] Registered new tserver with Master: bb0d89109f23435db76236a5f64b25d5 (127.18.196.194:36001)
Sep 25 02:57:23 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305043, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.253@KRBTEST.COM
I20260925 02:57:23.746580 25431 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.196.194@KRBTEST.COM'} at 127.18.196.194:45233
I20260925 02:57:23.753293 26556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:23.713358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:49479 (local address 127.18.196.253:44133)
0925 02:57:23.723837 (+ 10479us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:23.723844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:23.734011 (+ 10167us) server_negotiation.cc:407] Connection header received
0925 02:57:23.734271 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:23.734278 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:23.734372 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:23.734467 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:23.735975 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.737074 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.741472 (+ 4398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.742313 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.742474 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:23.747347 (+ 4873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:23.747354 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:23.747358 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:23.747401 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:23.750670 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:23.751497 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:23.751501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:23.751504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:23.751622 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:23.752040 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:23.752043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:23.752045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:23.752317 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:23.752434 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:23.752867 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 02:57:23.753045 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10330,"thread_start_us":3353,"threads_started":1}
I20260925 02:57:23.753582 26557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:23.716987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:44133 (local address 127.18.196.194:49479)
0925 02:57:23.733913 (+ 16926us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:23.733933 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:23.734113 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:23.734707 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:23.734711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:23.735370 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:23.735796 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:23.735808 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.737360 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.737365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:23.741317 (+ 3952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:23.741331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:23.742550 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:23.742557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:23.742762 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:23.743536 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:23.743557 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:23.747126 (+ 3569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:23.750823 (+ 3697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:23.750830 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:23.750857 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:23.751365 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:23.751731 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:23.751735 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:23.751739 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:23.751932 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:23.752438 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:23.752444 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:23.752607 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 02:57:23.752759 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 02:57:23.753132 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16798,"thread_start_us":14723,"threads_started":1}
I20260925 02:57:23.754390 26548 heartbeater.cc:347] Connected to a master server at 127.18.196.253:44133
I20260925 02:57:23.754546 26548 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:23.754999 26548 heartbeater.cc:511] Master 127.18.196.253:44133 requested a full tablet report, sending...
I20260925 02:57:23.756270 26035 ts_manager.cc:194] Registered new tserver with Master: bb0d89109f23435db76236a5f64b25d5 (127.18.196.194:36001)
I20260925 02:57:23.832494 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 25759
I20260925 02:57:23.849506 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:43605
--local_ip_for_outbound_sockets=127.18.196.195
--tserver_master_addrs=127.18.196.254:40775
--webserver_port=35019
--webserver_interface=127.18.196.195
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.195.keytab
--principal=oryx/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.196.254:40775,127.18.196.253:44133
--tserver_master_addrs=127.18.196.254:40775,127.18.196.253:44133,127.18.196.252:33819 with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:57:24.114710 26407 heartbeater.cc:503] Master 127.18.196.254:40775 was elected leader, sending a full tablet report...
W20260925 02:57:24.172928 26559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:24.173202 26559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:24.173295 26559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:24.181079 26559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:24.181191 26559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:24.181304 26559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 02:57:24.191192 26559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.195.keytab
--principal=oryx/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:43605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=35019
--tserver_master_addrs=127.18.196.254:40775,127.18.196.253:44133,127.18.196.252:33819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:24.192480 26559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:24.193748 26559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:24.207325 26568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:24.207394 26565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:24.207370 26566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:24.208554 26559 server_base.cc:1034] running on GCE node
I20260925 02:57:24.209203 26559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:24.210170 26559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:24.211383 26559 hybrid_clock.cc:648] HybridClock initialized: now 1790305044211326 us; error 46 us; skew 500 ppm
I20260925 02:57:24.211694 26559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:24 dist-test-slave-8zvk krb5kdc[25386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305044, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:24.214990 26559 init.cc:377] Logged in from keytab as oryx/127.18.196.195@KRBTEST.COM (short username oryx)
I20260925 02:57:24.216286 26559 webserver.cc:460] Webserver started at http://127.18.196.195:35019/ using document root <none> and password file <none>
I20260925 02:57:24.216897 26559 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:24.217013 26559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:24.222308 26559 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:57:24.224977 26575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:24.226325 26559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:57:24.226521 26559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "920f7345bb8940ff98552b67ad8fb652"
format_stamp: "Formatted at 2026-09-25 02:57:20 on dist-test-slave-8zvk"
server_key: "2e7dc84cf5c345c70ad755c62a25f687"
server_key_iv: "41c4ee872420e5fbf72cf58b5b8da6a1"
server_key_version: "encryptionkey@0"
I20260925 02:57:24.227597 26559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:24.242535 26559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:24.246384 26559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:24.246683 26559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:24.247684 26559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:24.249081 26559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:24.249189 26559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:24.249318 26559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:24.249425 26559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:24.283106 26559 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:43605
I20260925 02:57:24.283188 26687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:43605 every 8 connection(s)
I20260925 02:57:24.284534 26559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 02:57:24.289748 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 26559
Sep 25 02:57:24 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305044, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM
Sep 25 02:57:24 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305044, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.252@KRBTEST.COM
Sep 25 02:57:24 dist-test-slave-8zvk krb5kdc[25386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305044, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.253@KRBTEST.COM
I20260925 02:57:24.321236 26695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:24.288083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:33819 (local address 127.18.196.195:45699)
0925 02:57:24.289454 (+ 1371us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:24.289472 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:24.292789 (+ 3317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:24.298193 (+ 5404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:24.298198 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:24.298967 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:24.299423 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:24.299484 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.301787 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.301792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:24.303236 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:24.303257 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.304581 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.304588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:24.304800 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:24.305515 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:24.305539 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:24.309550 (+ 4011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:24.315014 (+ 5464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:24.315028 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:24.315047 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:24.315980 (+ 933us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:24.316517 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:24.316523 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:24.316528 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:24.316800 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:24.317533 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:24.317540 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:24.318124 (+ 584us) client_negotiation.cc:769] Sending connection context
0925 02:57:24.318790 (+ 666us) client_negotiation.cc:240] Negotiation successful
0925 02:57:24.319413 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1209,"mutex_wait_us":502,"thread_start_us":1251,"threads_started":1}
I20260925 02:57:24.322880 26692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:24.287327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:38033 (local address 127.18.196.254:40775)
0925 02:57:24.288591 (+ 1264us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:24.288601 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:24.289126 (+ 525us) server_negotiation.cc:407] Connection header received
0925 02:57:24.292728 (+ 3602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:24.292739 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:24.292856 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:24.293018 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:24.295663 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.297234 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.303425 (+ 6191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.303832 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.304002 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:24.309569 (+ 5567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:24.309578 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:24.309583 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:24.309650 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:24.314988 (+ 5338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:24.317261 (+ 2273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:24.317267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:24.317270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:24.317455 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:24.318639 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:24.318646 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:24.318652 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:24.319044 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:24.319203 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:24.322218 (+ 3015us) server_negotiation.cc:299] Negotiation successful
0925 02:57:24.322448 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1094,"thread_start_us":966,"threads_started":1}
I20260925 02:57:24.322897 26694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:24.288082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.18.196.195:38033)
0925 02:57:24.288970 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:24.289020 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:24.292324 (+ 3304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:24.293334 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:24.293351 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:24.294361 (+ 1010us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:24.295428 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:24.295464 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.297449 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.297454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:24.303255 (+ 5801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:24.303276 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.304273 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.304280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:24.304567 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:24.305353 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:24.305387 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:24.309289 (+ 3902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:24.315163 (+ 5874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:24.315169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:24.315174 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:24.316874 (+ 1700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:24.318098 (+ 1224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:24.318105 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:24.318111 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:24.318517 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:24.321738 (+ 3221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:24.321743 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:24.321922 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 02:57:24.322078 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 02:57:24.322378 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":2,"thread_start_us":506,"threads_started":1}
I20260925 02:57:24.325423 26689 heartbeater.cc:347] Connected to a master server at 127.18.196.254:40775
I20260925 02:57:24.325788 26689 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:24.326763 26689 heartbeater.cc:511] Master 127.18.196.254:40775 requested a full tablet report, sending...
I20260925 02:57:24.326953 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:24.289714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:44133 (local address 127.18.196.195:43477)
0925 02:57:24.300891 (+ 11177us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:24.300911 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:24.301028 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:24.302968 (+ 1940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:24.302973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:24.303597 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:24.303894 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:24.303905 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.306076 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.306081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:24.307621 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:24.307648 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.313681 (+ 6033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.313688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:24.314171 (+ 483us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:24.314780 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:24.314807 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:24.317934 (+ 3127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:24.322933 (+ 4999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:24.322938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:24.322941 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:24.323740 (+ 799us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:24.324461 (+ 721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:24.324469 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:24.324475 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:24.324670 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:24.325552 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:24.325560 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:24.325780 (+ 220us) client_negotiation.cc:769] Sending connection context
0925 02:57:24.325943 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 02:57:24.326324 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11034,"thread_start_us":10819,"threads_started":1}
I20260925 02:57:24.328472 25431 ts_manager.cc:194] Re-registered known tserver with Master: 920f7345bb8940ff98552b67ad8fb652 (127.18.196.195:43605)
I20260925 02:57:24.328572 26697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:24.289846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:43477 (local address 127.18.196.253:44133)
0925 02:57:24.301315 (+ 11469us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:24.301321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:24.301340 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:57:24.301417 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:24.301422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:24.301492 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:24.301592 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:24.304230 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.305838 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.307777 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.313165 (+ 5388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.313342 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:24.318422 (+ 5080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:24.318431 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:24.318439 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:24.318499 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:24.322744 (+ 4245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:24.323945 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:24.323953 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:24.323958 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:24.324280 (+ 322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:24.324819 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:24.324827 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:24.324833 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:24.325251 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:24.325473 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:24.327429 (+ 1956us) server_negotiation.cc:299] Negotiation successful
0925 02:57:24.328046 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11352,"thread_start_us":11268,"threads_started":1}
I20260925 02:57:24.329540 25431 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.196.195@KRBTEST.COM'} at 127.18.196.195:38033
I20260925 02:57:24.329450 26690 heartbeater.cc:347] Connected to a master server at 127.18.196.253:44133
I20260925 02:57:24.330477 26690 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:24.331045 26690 heartbeater.cc:511] Master 127.18.196.253:44133 requested a full tablet report, sending...
I20260925 02:57:24.331525 26693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:24.287316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:45699 (local address 127.18.196.252:33819)
0925 02:57:24.289266 (+ 1950us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:24.289272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:24.297819 (+ 8547us) server_negotiation.cc:407] Connection header received
0925 02:57:24.297906 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:24.297914 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:24.298038 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:24.298170 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:24.300576 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.301652 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.303715 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.304393 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.304548 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:24.310463 (+ 5915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:24.310471 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:24.310477 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:24.310527 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:24.314792 (+ 4265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:24.316185 (+ 1393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:24.316194 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:24.316198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:24.316358 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:24.316981 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:24.316986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:24.316990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:24.317359 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:24.330445 (+ 13086us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:24.330672 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 02:57:24.330950 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1716,"thread_start_us":1593,"threads_started":1}
I20260925 02:57:24.332641 26688 heartbeater.cc:347] Connected to a master server at 127.18.196.252:33819
I20260925 02:57:24.332764 26688 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:24.333467 26035 ts_manager.cc:194] Registered new tserver with Master: 920f7345bb8940ff98552b67ad8fb652 (127.18.196.195:43605)
I20260925 02:57:24.333575 26688 heartbeater.cc:511] Master 127.18.196.252:33819 requested a full tablet report, sending...
I20260925 02:57:24.334964 26225 ts_manager.cc:194] Registered new tserver with Master: 920f7345bb8940ff98552b67ad8fb652 (127.18.196.195:43605)
I20260925 02:57:24.342389 26705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:24.309993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39542 (local address 127.18.196.252:33819)
0925 02:57:24.315331 (+ 5338us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:24.315337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:24.317649 (+ 2312us) server_negotiation.cc:407] Connection header received
0925 02:57:24.317711 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:24.317715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:24.317789 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:24.317880 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:24.321808 (+ 3928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.323031 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.324879 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.325724 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.325852 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:24.335300 (+ 9448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:24.335308 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:24.335312 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:24.335348 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:24.339278 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:24.339946 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:24.339954 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:24.339959 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:24.340106 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:24.340512 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:24.340517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:24.340520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:24.340862 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:24.341102 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:24.341818 (+ 716us) server_negotiation.cc:299] Negotiation successful
0925 02:57:24.342038 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5154,"thread_start_us":2960,"threads_started":1}
I20260925 02:57:24.344177 26702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:24.306895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43352 (local address 127.18.196.254:40775)
0925 02:57:24.310792 (+ 3897us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:24.310799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:24.313766 (+ 2967us) server_negotiation.cc:407] Connection header received
0925 02:57:24.314049 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:24.314055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:24.314144 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:24.314235 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:24.326094 (+ 11859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.327229 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.329603 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.330394 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.330488 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:24.337399 (+ 6911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:24.337407 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:24.337413 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:24.337459 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:24.341212 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:24.341983 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:24.341988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:24.341991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:24.342106 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:24.342529 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:24.342537 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:24.342542 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:24.342896 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:24.343092 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:24.343454 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 02:57:24.343655 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3777,"thread_start_us":3949,"threads_started":1}
I20260925 02:57:24.352068 26697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:24.316139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.18.196.253:44133)
0925 02:57:24.334636 (+ 18497us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:24.334644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:24.335284 (+ 640us) server_negotiation.cc:407] Connection header received
0925 02:57:24.335557 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:24.335562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:24.335643 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:24.335706 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:24.338555 (+ 2849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.339761 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.341202 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.341688 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.341838 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:24.346157 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:24.346166 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:24.346173 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:24.346213 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:24.349417 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:24.350032 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:24.350039 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:24.350045 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:24.350196 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:24.350551 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:24.350556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:24.350559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:24.350953 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:24.351316 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:24.351481 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 02:57:24.351718 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18367,"thread_start_us":973,"threads_started":1}
I20260925 02:57:24.348323 25431 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:57:24.357499 25431 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:57:24.405117 26724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:24.378729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50694 (local address 127.18.196.194:36001)
0925 02:57:24.396085 (+ 17356us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:24.396094 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:24.396979 (+ 885us) server_negotiation.cc:407] Connection header received
0925 02:57:24.397185 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:24.397195 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:24.397451 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:24.397669 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:24.398410 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.400064 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.402958 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.404198 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.404365 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:24.404553 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:24.404697 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 02:57:24.404845 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17170,"thread_start_us":17353,"threads_started":1}
I20260925 02:57:24.405267 26723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:24.378447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:36001 (local address 127.0.0.1:50694)
0925 02:57:24.396793 (+ 18346us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:24.396812 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:24.396953 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:24.397659 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:24.397664 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:24.397690 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:24.398139 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:24.398158 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.400191 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.400195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:24.402733 (+ 2538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:24.402749 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.404365 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.404371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:24.404520 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:24.404550 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 02:57:24.404649 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 02:57:24.404767 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18186,"mutex_wait_us":1,"thread_start_us":17977,"threads_started":1}
I20260925 02:57:24.405701 26720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:24.376531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:38247 (local address 127.0.0.1:46302)
0925 02:57:24.393334 (+ 16803us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:24.393354 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:24.393500 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:24.394939 (+ 1439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:24.394948 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:24.394986 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:24.395445 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:24.395461 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.401125 (+ 5664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.401132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:24.403505 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:24.403524 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.404886 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.404892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:24.405053 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:24.405091 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 02:57:24.405177 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 02:57:24.405325 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16661,"thread_start_us":16515,"threads_started":1}
I20260925 02:57:24.406520 26721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:24.376788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46302 (local address 127.18.196.193:38247)
0925 02:57:24.393750 (+ 16962us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:24.393765 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:24.393804 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:57:24.393932 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:24.393944 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:24.394318 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:24.394600 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:24.396331 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.400923 (+ 4592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.403720 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.404657 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.404849 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:24.405018 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:24.405951 (+ 933us) server_negotiation.cc:299] Negotiation successful
0925 02:57:24.406147 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14180,"thread_start_us":14372,"threads_started":1}
I20260925 02:57:24.408596 26719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:24.376142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:43605 (local address 127.0.0.1:60566)
0925 02:57:24.383584 (+ 7442us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:24.383609 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:24.383752 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:24.398328 (+ 14576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:24.398333 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:24.398362 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:24.398711 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:24.398720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.400623 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.400629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:24.403273 (+ 2644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:24.403287 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.407839 (+ 4552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.407845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:24.408021 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:24.408051 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 02:57:24.408149 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 02:57:24.408272 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7267,"thread_start_us":7107,"threads_started":1}
I20260925 02:57:24.408722 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:24.376941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.18.196.195:43605)
0925 02:57:24.396142 (+ 19201us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:24.396150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:24.397741 (+ 1591us) server_negotiation.cc:407] Connection header received
0925 02:57:24.397865 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:24.397872 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:24.398119 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:24.398340 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:24.398906 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.400493 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.405102 (+ 4609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.406814 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.407012 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:24.407159 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:24.408217 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0925 02:57:24.408390 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19038,"thread_start_us":18837,"threads_started":1}
I20260925 02:57:24.409595 26481 tablet_service.cc:1467] Processing CreateTablet for tablet cf18d53c5df14ee5b3bd512e3fc9f538 (DEFAULT_TABLE table=test-table [id=b7f225430ad346f9bd51855da57d8682]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:57:24.409991 26341 tablet_service.cc:1467] Processing CreateTablet for tablet cf18d53c5df14ee5b3bd512e3fc9f538 (DEFAULT_TABLE table=test-table [id=b7f225430ad346f9bd51855da57d8682]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:57:24.411233 26481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf18d53c5df14ee5b3bd512e3fc9f538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:24.411464 26341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf18d53c5df14ee5b3bd512e3fc9f538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:24.414507 26623 tablet_service.cc:1467] Processing CreateTablet for tablet cf18d53c5df14ee5b3bd512e3fc9f538 (DEFAULT_TABLE table=test-table [id=b7f225430ad346f9bd51855da57d8682]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:57:24.416083 26623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf18d53c5df14ee5b3bd512e3fc9f538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:24.424628 26725 tablet_bootstrap.cc:492] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618: Bootstrap starting.
I20260925 02:57:24.428035 26726 tablet_bootstrap.cc:492] T cf18d53c5df14ee5b3bd512e3fc9f538 P bb0d89109f23435db76236a5f64b25d5: Bootstrap starting.
I20260925 02:57:24.428431 26725 tablet_bootstrap.cc:654] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:24.430311 26725 log.cc:824] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:24.432938 26726 tablet_bootstrap.cc:654] T cf18d53c5df14ee5b3bd512e3fc9f538 P bb0d89109f23435db76236a5f64b25d5: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:24.434996 26726 log.cc:824] T cf18d53c5df14ee5b3bd512e3fc9f538 P bb0d89109f23435db76236a5f64b25d5: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:24.435971 26728 tablet_bootstrap.cc:492] T cf18d53c5df14ee5b3bd512e3fc9f538 P 920f7345bb8940ff98552b67ad8fb652: Bootstrap starting.
I20260925 02:57:24.440439 26728 tablet_bootstrap.cc:654] T cf18d53c5df14ee5b3bd512e3fc9f538 P 920f7345bb8940ff98552b67ad8fb652: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:24.440785 26725 tablet_bootstrap.cc:492] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618: No bootstrap required, opened a new log
I20260925 02:57:24.441119 26725 ts_tablet_manager.cc:1397] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20260925 02:57:24.441627 26726 tablet_bootstrap.cc:492] T cf18d53c5df14ee5b3bd512e3fc9f538 P bb0d89109f23435db76236a5f64b25d5: No bootstrap required, opened a new log
I20260925 02:57:24.441944 26728 log.cc:824] T cf18d53c5df14ee5b3bd512e3fc9f538 P 920f7345bb8940ff98552b67ad8fb652: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:24.442050 26726 ts_tablet_manager.cc:1397] T cf18d53c5df14ee5b3bd512e3fc9f538 P bb0d89109f23435db76236a5f64b25d5: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260925 02:57:24.449931 26728 tablet_bootstrap.cc:492] T cf18d53c5df14ee5b3bd512e3fc9f538 P 920f7345bb8940ff98552b67ad8fb652: No bootstrap required, opened a new log
I20260925 02:57:24.449790 26725 raft_consensus.cc:348] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920f7345bb8940ff98552b67ad8fb652" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43605 } } peers { permanent_uuid: "c2fa8d55ea204f48a127e243690cb618" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38247 } } peers { permanent_uuid: "bb0d89109f23435db76236a5f64b25d5" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 36001 } }
I20260925 02:57:24.450239 26728 ts_tablet_manager.cc:1397] T cf18d53c5df14ee5b3bd512e3fc9f538 P 920f7345bb8940ff98552b67ad8fb652: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260925 02:57:24.450261 26725 raft_consensus.cc:374] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:24.450358 26725 raft_consensus.cc:729] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2fa8d55ea204f48a127e243690cb618, State: Initialized, Role: FOLLOWER
I20260925 02:57:24.451057 26725 consensus_queue.cc:260] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920f7345bb8940ff98552b67ad8fb652" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43605 } } peers { permanent_uuid: "c2fa8d55ea204f48a127e243690cb618" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38247 } } peers { permanent_uuid: "bb0d89109f23435db76236a5f64b25d5" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 36001 } }
I20260925 02:57:24.452896 26725 ts_tablet_manager.cc:1428] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260925 02:57:24.454797 26726 raft_consensus.cc:348] T cf18d53c5df14ee5b3bd512e3fc9f538 P bb0d89109f23435db76236a5f64b25d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920f7345bb8940ff98552b67ad8fb652" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43605 } } peers { permanent_uuid: "c2fa8d55ea204f48a127e243690cb618" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38247 } } peers { permanent_uuid: "bb0d89109f23435db76236a5f64b25d5" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 36001 } }
I20260925 02:57:24.455322 26726 raft_consensus.cc:374] T cf18d53c5df14ee5b3bd512e3fc9f538 P bb0d89109f23435db76236a5f64b25d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:24.455555 26726 raft_consensus.cc:729] T cf18d53c5df14ee5b3bd512e3fc9f538 P bb0d89109f23435db76236a5f64b25d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb0d89109f23435db76236a5f64b25d5, State: Initialized, Role: FOLLOWER
I20260925 02:57:24.456316 26726 consensus_queue.cc:260] T cf18d53c5df14ee5b3bd512e3fc9f538 P bb0d89109f23435db76236a5f64b25d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920f7345bb8940ff98552b67ad8fb652" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43605 } } peers { permanent_uuid: "c2fa8d55ea204f48a127e243690cb618" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38247 } } peers { permanent_uuid: "bb0d89109f23435db76236a5f64b25d5" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 36001 } }
I20260925 02:57:24.460122 26726 ts_tablet_manager.cc:1428] T cf18d53c5df14ee5b3bd512e3fc9f538 P bb0d89109f23435db76236a5f64b25d5: Time spent starting tablet: real 0.018s user 0.018s sys 0.000s
I20260925 02:57:24.460476 26547 heartbeater.cc:503] Master 127.18.196.254:40775 was elected leader, sending a full tablet report...
W20260925 02:57:24.462770 26549 tablet.cc:2369] T cf18d53c5df14ee5b3bd512e3fc9f538 P bb0d89109f23435db76236a5f64b25d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:57:24.465044 26728 raft_consensus.cc:348] T cf18d53c5df14ee5b3bd512e3fc9f538 P 920f7345bb8940ff98552b67ad8fb652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920f7345bb8940ff98552b67ad8fb652" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43605 } } peers { permanent_uuid: "c2fa8d55ea204f48a127e243690cb618" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38247 } } peers { permanent_uuid: "bb0d89109f23435db76236a5f64b25d5" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 36001 } }
I20260925 02:57:24.465406 26728 raft_consensus.cc:374] T cf18d53c5df14ee5b3bd512e3fc9f538 P 920f7345bb8940ff98552b67ad8fb652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:24.465512 26728 raft_consensus.cc:729] T cf18d53c5df14ee5b3bd512e3fc9f538 P 920f7345bb8940ff98552b67ad8fb652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 920f7345bb8940ff98552b67ad8fb652, State: Initialized, Role: FOLLOWER
I20260925 02:57:24.466049 26728 consensus_queue.cc:260] T cf18d53c5df14ee5b3bd512e3fc9f538 P 920f7345bb8940ff98552b67ad8fb652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920f7345bb8940ff98552b67ad8fb652" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43605 } } peers { permanent_uuid: "c2fa8d55ea204f48a127e243690cb618" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38247 } } peers { permanent_uuid: "bb0d89109f23435db76236a5f64b25d5" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 36001 } }
I20260925 02:57:24.467085 26689 heartbeater.cc:503] Master 127.18.196.254:40775 was elected leader, sending a full tablet report...
I20260925 02:57:24.467717 26728 ts_tablet_manager.cc:1428] T cf18d53c5df14ee5b3bd512e3fc9f538 P 920f7345bb8940ff98552b67ad8fb652: Time spent starting tablet: real 0.017s user 0.004s sys 0.009s
W20260925 02:57:24.537364 26691 tablet.cc:2369] T cf18d53c5df14ee5b3bd512e3fc9f538 P 920f7345bb8940ff98552b67ad8fb652: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:57:24.564256 26409 tablet.cc:2369] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:57:24.622653 26731 raft_consensus.cc:482] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:57:24.622992 26731 raft_consensus.cc:504] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920f7345bb8940ff98552b67ad8fb652" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43605 } } peers { permanent_uuid: "c2fa8d55ea204f48a127e243690cb618" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38247 } } peers { permanent_uuid: "bb0d89109f23435db76236a5f64b25d5" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 36001 } }
I20260925 02:57:24.625118 26731 leader_election.cc:288] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 920f7345bb8940ff98552b67ad8fb652 (127.18.196.195:43605), bb0d89109f23435db76236a5f64b25d5 (127.18.196.194:36001)
I20260925 02:57:24.632954 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:24.625449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:58653 (local address 127.18.196.195:43605)
0925 02:57:24.625699 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:24.625706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:24.627272 (+ 1566us) server_negotiation.cc:407] Connection header received
0925 02:57:24.627501 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:24.627506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:24.627573 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:24.627668 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:24.628361 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.629175 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.631106 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.631968 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.632142 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:24.632224 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:24.632555 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 02:57:24.632679 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260925 02:57:24.633189 26724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:24.625480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:40235 (local address 127.18.196.194:36001)
0925 02:57:24.625767 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:24.625775 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:24.627456 (+ 1681us) server_negotiation.cc:407] Connection header received
0925 02:57:24.627687 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:24.627691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:24.627747 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:24.627833 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:24.628419 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.629115 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.631076 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.632032 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.632195 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:24.632274 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:24.632604 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 02:57:24.632729 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:57:24.633239 26734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:24.625462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:43605 (local address 127.18.196.193:58653)
0925 02:57:24.627155 (+ 1693us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:24.627171 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:24.627344 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:24.627684 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:24.627688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:24.627713 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:24.628217 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:24.628230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.629358 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.629362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:24.630955 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:24.630967 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.632204 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.632210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:24.632393 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:24.632422 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 02:57:24.632526 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 02:57:24.632674 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1470,"mutex_wait_us":23,"spinlock_wait_cycles":1792,"thread_start_us":1042,"threads_started":1}
I20260925 02:57:24.633617 26735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:24.626110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:36001 (local address 127.18.196.193:40235)
0925 02:57:24.627351 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:24.627365 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:24.627545 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:24.627863 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:24.627867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:24.627905 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:24.628240 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:24.628248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.629249 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.629254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:24.630937 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:24.630948 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.632255 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.632261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:24.632450 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:24.632473 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:57:24.632575 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 02:57:24.632720 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1130,"mutex_wait_us":63,"spinlock_wait_cycles":1920,"thread_start_us":839,"threads_started":1}
I20260925 02:57:24.634357 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf18d53c5df14ee5b3bd512e3fc9f538" candidate_uuid: "c2fa8d55ea204f48a127e243690cb618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "920f7345bb8940ff98552b67ad8fb652" is_pre_election: true
I20260925 02:57:24.634684 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf18d53c5df14ee5b3bd512e3fc9f538" candidate_uuid: "c2fa8d55ea204f48a127e243690cb618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb0d89109f23435db76236a5f64b25d5" is_pre_election: true
I20260925 02:57:24.634883 26643 raft_consensus.cc:2454] T cf18d53c5df14ee5b3bd512e3fc9f538 P 920f7345bb8940ff98552b67ad8fb652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2fa8d55ea204f48a127e243690cb618 in term 0.
I20260925 02:57:24.635231 26501 raft_consensus.cc:2454] T cf18d53c5df14ee5b3bd512e3fc9f538 P bb0d89109f23435db76236a5f64b25d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2fa8d55ea204f48a127e243690cb618 in term 0.
I20260925 02:57:24.635596 26293 leader_election.cc:302] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 920f7345bb8940ff98552b67ad8fb652, c2fa8d55ea204f48a127e243690cb618; no voters:
I20260925 02:57:24.635980 26731 raft_consensus.cc:2781] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:57:24.636274 26731 raft_consensus.cc:482] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:57:24.636401 26731 raft_consensus.cc:3037] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:24.638556 26731 raft_consensus.cc:504] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920f7345bb8940ff98552b67ad8fb652" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43605 } } peers { permanent_uuid: "c2fa8d55ea204f48a127e243690cb618" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38247 } } peers { permanent_uuid: "bb0d89109f23435db76236a5f64b25d5" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 36001 } }
I20260925 02:57:24.639389 26731 leader_election.cc:288] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618 [CANDIDATE]: Term 1 election: Requested vote from peers 920f7345bb8940ff98552b67ad8fb652 (127.18.196.195:43605), bb0d89109f23435db76236a5f64b25d5 (127.18.196.194:36001)
I20260925 02:57:24.639659 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf18d53c5df14ee5b3bd512e3fc9f538" candidate_uuid: "c2fa8d55ea204f48a127e243690cb618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "920f7345bb8940ff98552b67ad8fb652"
I20260925 02:57:24.639863 26643 raft_consensus.cc:3037] T cf18d53c5df14ee5b3bd512e3fc9f538 P 920f7345bb8940ff98552b67ad8fb652 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:24.639889 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf18d53c5df14ee5b3bd512e3fc9f538" candidate_uuid: "c2fa8d55ea204f48a127e243690cb618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb0d89109f23435db76236a5f64b25d5"
I20260925 02:57:24.640177 26501 raft_consensus.cc:3037] T cf18d53c5df14ee5b3bd512e3fc9f538 P bb0d89109f23435db76236a5f64b25d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:24.642112 26643 raft_consensus.cc:2454] T cf18d53c5df14ee5b3bd512e3fc9f538 P 920f7345bb8940ff98552b67ad8fb652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2fa8d55ea204f48a127e243690cb618 in term 1.
I20260925 02:57:24.642531 26293 leader_election.cc:302] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 920f7345bb8940ff98552b67ad8fb652, c2fa8d55ea204f48a127e243690cb618; no voters:
I20260925 02:57:24.642722 26501 raft_consensus.cc:2454] T cf18d53c5df14ee5b3bd512e3fc9f538 P bb0d89109f23435db76236a5f64b25d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2fa8d55ea204f48a127e243690cb618 in term 1.
I20260925 02:57:24.643016 26731 raft_consensus.cc:2781] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:24.643493 26731 raft_consensus.cc:686] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618 [term 1 LEADER]: Becoming Leader. State: Replica: c2fa8d55ea204f48a127e243690cb618, State: Running, Role: LEADER
I20260925 02:57:24.644016 26731 consensus_queue.cc:237] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920f7345bb8940ff98552b67ad8fb652" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43605 } } peers { permanent_uuid: "c2fa8d55ea204f48a127e243690cb618" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38247 } } peers { permanent_uuid: "bb0d89109f23435db76236a5f64b25d5" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 36001 } }
I20260925 02:57:24.649223 25431 catalog_manager.cc:5495] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2fa8d55ea204f48a127e243690cb618 (127.18.196.193). New cstate: current_term: 1 leader_uuid: "c2fa8d55ea204f48a127e243690cb618" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920f7345bb8940ff98552b67ad8fb652" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2fa8d55ea204f48a127e243690cb618" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb0d89109f23435db76236a5f64b25d5" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 36001 } health_report { overall_health: UNKNOWN } } }
I20260925 02:57:24.687817 26721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:24.681168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46312 (local address 127.18.196.193:38247)
0925 02:57:24.681478 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:24.681483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:24.681504 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:57:24.681706 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:24.681711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:24.681829 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:24.681998 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:57:24.682477 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.683681 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.685259 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.685903 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.686018 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:24.686519 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:57:24.687186 (+ 667us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:57:24.687307 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:24.687425 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 02:57:24.687506 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260925 02:57:24.697784 26501 raft_consensus.cc:1261] T cf18d53c5df14ee5b3bd512e3fc9f538 P bb0d89109f23435db76236a5f64b25d5 [term 1 FOLLOWER]: Refusing update from remote peer c2fa8d55ea204f48a127e243690cb618: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:57:24.698062 26643 raft_consensus.cc:1261] T cf18d53c5df14ee5b3bd512e3fc9f538 P 920f7345bb8940ff98552b67ad8fb652 [term 1 FOLLOWER]: Refusing update from remote peer c2fa8d55ea204f48a127e243690cb618: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:57:24.699220 26736 consensus_queue.cc:1035] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618 [LEADER]: Connected to new peer: Peer: permanent_uuid: "920f7345bb8940ff98552b67ad8fb652" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:57:24.699769 26731 consensus_queue.cc:1035] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb0d89109f23435db76236a5f64b25d5" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:57:24.714601 26739 mvcc.cc:205] Tried to move back new op lower bound from 7333089463071834112 to 7333089462865403904. Current Snapshot: MvccSnapshot[applied={T|T < 7333089463071834112}]
I20260925 02:57:24.736195 25430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:57:24.736582 25430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:57:24.747496 25430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d736dfd4e08d41148d0bbc56465191ca: Sending DeleteTablet for 3 replicas of tablet cf18d53c5df14ee5b3bd512e3fc9f538
I20260925 02:57:24.748916 26623 tablet_service.cc:1514] Processing DeleteTablet for tablet cf18d53c5df14ee5b3bd512e3fc9f538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:24 UTC) from {username='oryx', principal='oryx/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:60566
I20260925 02:57:24.749079 26341 tablet_service.cc:1514] Processing DeleteTablet for tablet cf18d53c5df14ee5b3bd512e3fc9f538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:24 UTC) from {username='oryx', principal='oryx/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:46302
I20260925 02:57:24.750603 26481 tablet_service.cc:1514] Processing DeleteTablet for tablet cf18d53c5df14ee5b3bd512e3fc9f538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:24 UTC) from {username='oryx', principal='oryx/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:50694
I20260925 02:57:24.764488 26750 tablet_replica.cc:330] T cf18d53c5df14ee5b3bd512e3fc9f538 P 920f7345bb8940ff98552b67ad8fb652: stopping tablet replica
I20260925 02:57:24.764979 26750 raft_consensus.cc:2229] T cf18d53c5df14ee5b3bd512e3fc9f538 P 920f7345bb8940ff98552b67ad8fb652 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:57:24.765538 26750 raft_consensus.cc:2258] T cf18d53c5df14ee5b3bd512e3fc9f538 P 920f7345bb8940ff98552b67ad8fb652 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:57:24.766470 26751 tablet_replica.cc:330] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618: stopping tablet replica
I20260925 02:57:24.767849 26750 ts_tablet_manager.cc:1905] T cf18d53c5df14ee5b3bd512e3fc9f538 P 920f7345bb8940ff98552b67ad8fb652: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:57:24.768230 26751 raft_consensus.cc:2229] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:57:24.769274 26751 raft_consensus.cc:2258] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:57:24.771652 26752 tablet_replica.cc:330] T cf18d53c5df14ee5b3bd512e3fc9f538 P bb0d89109f23435db76236a5f64b25d5: stopping tablet replica
I20260925 02:57:24.772425 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 25399
I20260925 02:57:24.773279 26752 raft_consensus.cc:2229] T cf18d53c5df14ee5b3bd512e3fc9f538 P bb0d89109f23435db76236a5f64b25d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:57:24.773924 26752 raft_consensus.cc:2258] T cf18d53c5df14ee5b3bd512e3fc9f538 P bb0d89109f23435db76236a5f64b25d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:57:24.775293 25461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:24.776615 26752 ts_tablet_manager.cc:1905] T cf18d53c5df14ee5b3bd512e3fc9f538 P bb0d89109f23435db76236a5f64b25d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:57:24.783306 26750 ts_tablet_manager.cc:1918] T cf18d53c5df14ee5b3bd512e3fc9f538 P 920f7345bb8940ff98552b67ad8fb652: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:57:24.785244 26750 log.cc:1196] T cf18d53c5df14ee5b3bd512e3fc9f538 P 920f7345bb8940ff98552b67ad8fb652: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/cf18d53c5df14ee5b3bd512e3fc9f538
I20260925 02:57:24.786070 26750 ts_tablet_manager.cc:1939] T cf18d53c5df14ee5b3bd512e3fc9f538 P 920f7345bb8940ff98552b67ad8fb652: Deleting consensus metadata
I20260925 02:57:24.786501 26751 ts_tablet_manager.cc:1905] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:57:24.797843 26752 ts_tablet_manager.cc:1918] T cf18d53c5df14ee5b3bd512e3fc9f538 P bb0d89109f23435db76236a5f64b25d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:57:24.798195 26752 log.cc:1196] T cf18d53c5df14ee5b3bd512e3fc9f538 P bb0d89109f23435db76236a5f64b25d5: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/cf18d53c5df14ee5b3bd512e3fc9f538
I20260925 02:57:24.798945 26752 ts_tablet_manager.cc:1939] T cf18d53c5df14ee5b3bd512e3fc9f538 P bb0d89109f23435db76236a5f64b25d5: Deleting consensus metadata
I20260925 02:57:24.800379 26751 ts_tablet_manager.cc:1918] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:57:24.801795 26751 log.cc:1196] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/wal/wals/cf18d53c5df14ee5b3bd512e3fc9f538
I20260925 02:57:24.803117 26751 ts_tablet_manager.cc:1939] T cf18d53c5df14ee5b3bd512e3fc9f538 P c2fa8d55ea204f48a127e243690cb618: Deleting consensus metadata
I20260925 02:57:24.805763 26697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:24.753124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47290 (local address 127.18.196.253:44133)
0925 02:57:24.753340 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:24.753349 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:24.770926 (+ 17577us) server_negotiation.cc:407] Connection header received
0925 02:57:24.771016 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:24.771022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:24.771111 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:24.771203 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:24.774373 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.775484 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.778602 (+ 3118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.779645 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.779779 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:24.785214 (+ 5435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:24.785220 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:24.785224 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:24.785264 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:24.802349 (+ 17085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:24.803286 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:24.803292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:24.803296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:24.803455 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:24.803897 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:24.803902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:24.803905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:24.804279 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:24.804528 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:24.805190 (+ 662us) server_negotiation.cc:299] Negotiation successful
0925 02:57:24.805406 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:57:24.824347 26705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:24.770579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39558 (local address 127.18.196.252:33819)
0925 02:57:24.770809 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:24.770817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:24.807602 (+ 36785us) server_negotiation.cc:407] Connection header received
0925 02:57:24.807832 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:24.807837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:24.807941 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:24.808004 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:24.809823 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.810794 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.812774 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:24.813379 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:24.813532 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:24.817518 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:24.817523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:24.817526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:24.817558 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:24.821153 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:24.821954 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:24.821959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:24.821963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:24.822104 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:24.822503 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:24.822508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:24.822512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:24.822900 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:24.823181 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:24.823780 (+ 599us) server_negotiation.cc:299] Negotiation successful
0925 02:57:24.824011 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 02:57:24.956418 25415 catalog_manager.cc:4841] TS c2fa8d55ea204f48a127e243690cb618 (127.18.196.193:38247): tablet cf18d53c5df14ee5b3bd512e3fc9f538 (table test-table [id=b7f225430ad346f9bd51855da57d8682]) successfully deleted
I20260925 02:57:24.956588 25418 catalog_manager.cc:4841] TS bb0d89109f23435db76236a5f64b25d5 (127.18.196.194:36001): tablet cf18d53c5df14ee5b3bd512e3fc9f538 (table test-table [id=b7f225430ad346f9bd51855da57d8682]) successfully deleted
I20260925 02:57:24.957716 25416 catalog_manager.cc:4841] TS 920f7345bb8940ff98552b67ad8fb652 (127.18.196.195:43605): tablet cf18d53c5df14ee5b3bd512e3fc9f538 (table test-table [id=b7f225430ad346f9bd51855da57d8682]) successfully deleted
W20260925 02:57:24.958045 25416 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.18.196.195:43605, user_credentials={real_user=slave}} blocked reactor thread for 169660us
I20260925 02:57:24.959270 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 25399
I20260925 02:57:25.015851 26697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:24.999852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47294 (local address 127.18.196.253:44133)
0925 02:57:25.000058 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:25.000064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:25.000160 (+ 96us) server_negotiation.cc:407] Connection header received
0925 02:57:25.000347 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:25.000353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:25.000435 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:25.000504 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:25.002158 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:25.003323 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:25.004951 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:25.005392 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:25.005555 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:25.009213 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:25.009220 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:25.009226 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:25.009271 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:25.013068 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:25.013879 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:25.013885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:25.013888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:25.014033 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:25.014374 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:25.014380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:25.014383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:25.014703 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:25.014984 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:25.015307 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 02:57:25.015533 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 02:57:25.016594 26705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:24.997836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39566 (local address 127.18.196.252:33819)
0925 02:57:24.998083 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:24.998092 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:24.999662 (+ 1570us) server_negotiation.cc:407] Connection header received
0925 02:57:24.999996 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:25.000001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:25.000089 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:25.000205 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:25.002539 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:25.003713 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:25.005635 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:25.006278 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:25.006409 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:25.009538 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:25.009543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:25.009546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:25.009586 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:25.012860 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:25.013776 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:25.013781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:25.013787 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:25.013958 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:25.014920 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:25.014925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:25.014928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:25.015267 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:25.015531 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:25.016047 (+ 516us) server_negotiation.cc:299] Negotiation successful
0925 02:57:25.016263 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 02:57:25.731630 26761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:25.730746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.18.196.193:56741)
0925 02:57:25.731272 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:25.731437 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":317,"threads_started":1}
W20260925 02:57:25.732124 26407 heartbeater.cc:644] Failed to heartbeat to 127.18.196.254:40775 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
I20260925 02:57:25.761303 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:25.760418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.18.196.194:43545)
0925 02:57:25.760944 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:25.761100 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":260,"threads_started":1}
W20260925 02:57:25.761811 26547 heartbeater.cc:644] Failed to heartbeat to 127.18.196.254:40775 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
I20260925 02:57:25.764029 26763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:25.763408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.18.196.195:34473)
0925 02:57:25.763769 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:25.763872 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":180,"threads_started":1}
W20260925 02:57:25.764395 26689 heartbeater.cc:644] Failed to heartbeat to 127.18.196.254:40775 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
I20260925 02:57:26.246366 26764 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d736dfd4e08d41148d0bbc56465191ca)
I20260925 02:57:26.246642 26765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d736dfd4e08d41148d0bbc56465191ca)
I20260925 02:57:26.246652 26764 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: false } }
I20260925 02:57:26.246915 26765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: false } }
I20260925 02:57:26.248112 26765 leader_election.cc:288] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d736dfd4e08d41148d0bbc56465191ca (127.18.196.254:40775), ab6a987abd184695be36cdde2d06d24e (127.18.196.253:44133)
I20260925 02:57:26.249013 26764 leader_election.cc:288] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d736dfd4e08d41148d0bbc56465191ca (127.18.196.254:40775), 469b93b3d2084e29807a456e82a08b40 (127.18.196.252:33819)
I20260925 02:57:26.249388 26766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.248158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.0.0.1:43802)
0925 02:57:26.248776 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.249029 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":279,"threads_started":1}
W20260925 02:57:26.249656 26020 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
I20260925 02:57:26.250097 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.248193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.0.0.1:43816)
0925 02:57:26.249625 (+ 1432us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.249820 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1302,"thread_start_us":1149,"threads_started":1}
I20260925 02:57:26.253111 26766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.252712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.0.0.1:43832)
0925 02:57:26.252879 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.252962 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 02:57:26.253211 26210 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
W20260925 02:57:26.253278 26020 leader_election.cc:334] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d736dfd4e08d41148d0bbc56465191ca (127.18.196.254:40775): Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
I20260925 02:57:26.255246 26769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.254595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.0.0.1:43836)
0925 02:57:26.254777 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.254995 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 02:57:26.255448 26210 leader_election.cc:334] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d736dfd4e08d41148d0bbc56465191ca (127.18.196.254:40775): Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
I20260925 02:57:26.257841 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.249832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:44133 (local address 127.0.0.1:47306)
0925 02:57:26.250507 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.250530 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:26.250673 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:26.251465 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:26.251469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:26.251498 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:26.251733 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:26.251750 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.253363 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.253367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:26.255679 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:26.255693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.257332 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.257336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:26.257466 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:26.257491 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 02:57:26.257565 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 02:57:26.257664 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":2356,"threads_started":1}
I20260925 02:57:26.258111 26770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.249794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47306 (local address 127.18.196.253:44133)
0925 02:57:26.251193 (+ 1399us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:26.251201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:26.251217 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:57:26.251277 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:26.251280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:26.251331 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:26.251448 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:26.251955 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.253185 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.255838 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.257171 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.257333 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:26.257411 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:26.257613 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 02:57:26.257738 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1287,"thread_start_us":1105,"threads_started":1}
I20260925 02:57:26.258671 26045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "469b93b3d2084e29807a456e82a08b40" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ab6a987abd184695be36cdde2d06d24e" is_pre_election: true
I20260925 02:57:26.258999 26045 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 469b93b3d2084e29807a456e82a08b40 in term 1.
I20260925 02:57:26.259508 26210 leader_election.cc:302] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 469b93b3d2084e29807a456e82a08b40, ab6a987abd184695be36cdde2d06d24e; no voters: d736dfd4e08d41148d0bbc56465191ca
I20260925 02:57:26.259877 26765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 02:57:26.260046 26767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.248232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:33819 (local address 127.0.0.1:39578)
0925 02:57:26.249076 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.249095 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:26.249220 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:26.254166 (+ 4946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:26.254170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:26.254191 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:26.254536 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:26.254551 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.255949 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.255957 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:26.258012 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:26.258030 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.259288 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.259294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:26.259455 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:26.259499 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 02:57:26.259601 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 02:57:26.259728 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":569,"threads_started":1}
I20260925 02:57:26.260037 26765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d736dfd4e08d41148d0bbc56465191ca)
I20260925 02:57:26.260140 26765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:57:26.260218 26772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.252408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39578 (local address 127.18.196.252:33819)
0925 02:57:26.253575 (+ 1167us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:26.253586 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:26.253606 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:57:26.253675 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:26.253687 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:26.253785 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:26.253917 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:26.254780 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.255822 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.258203 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.259130 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.259300 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:26.259369 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:26.259621 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 02:57:26.259790 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":720,"thread_start_us":572,"threads_started":1}
I20260925 02:57:26.260756 26235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ab6a987abd184695be36cdde2d06d24e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "469b93b3d2084e29807a456e82a08b40" is_pre_election: true
I20260925 02:57:26.262619 26765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: false } }
I20260925 02:57:26.263087 26235 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab6a987abd184695be36cdde2d06d24e in current term 2: Already voted for candidate 469b93b3d2084e29807a456e82a08b40 in this term.
I20260925 02:57:26.263358 26765 leader_election.cc:288] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [CANDIDATE]: Term 2 election: Requested vote from peers d736dfd4e08d41148d0bbc56465191ca (127.18.196.254:40775), ab6a987abd184695be36cdde2d06d24e (127.18.196.253:44133)
I20260925 02:57:26.263945 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.263433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.0.0.1:43848)
0925 02:57:26.263643 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.263741 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260925 02:57:26.264065 26021 leader_election.cc:302] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab6a987abd184695be36cdde2d06d24e; no voters: 469b93b3d2084e29807a456e82a08b40, d736dfd4e08d41148d0bbc56465191ca
I20260925 02:57:26.264137 26045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "469b93b3d2084e29807a456e82a08b40" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ab6a987abd184695be36cdde2d06d24e"
I20260925 02:57:26.264405 26045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:57:26.265476 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.265103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.0.0.1:43858)
0925 02:57:26.265262 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.265363 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 02:57:26.265635 26210 leader_election.cc:334] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d736dfd4e08d41148d0bbc56465191ca (127.18.196.254:40775): Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
I20260925 02:57:26.266563 26045 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 469b93b3d2084e29807a456e82a08b40 in term 2.
I20260925 02:57:26.266742 26764 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:57:26.266995 26210 leader_election.cc:302] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 469b93b3d2084e29807a456e82a08b40, ab6a987abd184695be36cdde2d06d24e; no voters: d736dfd4e08d41148d0bbc56465191ca
I20260925 02:57:26.267244 26765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:57:26.267650 26765 raft_consensus.cc:686] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [term 2 LEADER]: Becoming Leader. State: Replica: 469b93b3d2084e29807a456e82a08b40, State: Running, Role: LEADER
I20260925 02:57:26.268023 26765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: false } }
I20260925 02:57:26.269616 26774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 469b93b3d2084e29807a456e82a08b40. Latest consensus state: current_term: 2 leader_uuid: "469b93b3d2084e29807a456e82a08b40" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: false } } }
I20260925 02:57:26.270056 26774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:26.270702 26776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:26.276284 26776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:26.277122 26776 catalog_manager.cc:1259] Loaded cluster ID: 81e88c27e36e498f87565d50facd8f55
I20260925 02:57:26.277196 26776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:26.278322 26776 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:26.279181 26776 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40: Loaded TSK: 0
I20260925 02:57:26.280182 26776 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:57:26.299067 26225 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:57:26.305769 26225 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:57:26.309515 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.309085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.0.0.1:43874)
0925 02:57:26.309267 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.309354 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 02:57:26.309899 26045 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [term 2 FOLLOWER]: Refusing update from remote peer 469b93b3d2084e29807a456e82a08b40: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 02:57:26.310470 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.310130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.0.0.1:43888)
0925 02:57:26.310276 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.310351 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 02:57:26.310652 26210 consensus_peers.cc:487] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 -> Peer d736dfd4e08d41148d0bbc56465191ca (127.18.196.254:40775): Couldn't send request to peer d736dfd4e08d41148d0bbc56465191ca. Status: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 02:57:26.310704 26774 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 02:57:26.313205 26764 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 469b93b3d2084e29807a456e82a08b40. Latest consensus state: current_term: 2 leader_uuid: "469b93b3d2084e29807a456e82a08b40" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: false } } }
I20260925 02:57:26.313534 26764 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:57:26.315176 26765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "469b93b3d2084e29807a456e82a08b40" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: false } } }
I20260925 02:57:26.315536 26765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40 [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:26.320809 26778 mvcc.cc:205] Tried to move back new op lower bound from 7333089469678075904 to 7333089469517225984. Current Snapshot: MvccSnapshot[applied={T|T < 7333089469678075904}]
I20260925 02:57:26.323359 26764 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "469b93b3d2084e29807a456e82a08b40" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d736dfd4e08d41148d0bbc56465191ca" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40775 } } peers { permanent_uuid: "ab6a987abd184695be36cdde2d06d24e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 44133 } attrs { promote: false } } peers { permanent_uuid: "469b93b3d2084e29807a456e82a08b40" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 33819 } attrs { promote: false } } }
I20260925 02:57:26.323800 26764 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab6a987abd184695be36cdde2d06d24e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:57:26.335711 26783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.328372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46320 (local address 127.18.196.193:38247)
0925 02:57:26.328944 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:26.328950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:26.328968 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:57:26.329038 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:26.329044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:26.329119 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:26.329712 (+ 593us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:26.329913 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.331150 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.333506 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.334722 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.334908 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:26.334999 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:26.335261 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 02:57:26.335396 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":297,"threads_started":1}
I20260925 02:57:26.335711 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.328261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:38247 (local address 127.0.0.1:46320)
0925 02:57:26.328454 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.328472 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:26.328615 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:26.329300 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:26.329309 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:26.329335 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:26.329557 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:26.329570 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.331323 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.331328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:26.333316 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:26.333330 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.334900 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.334906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:26.335105 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:26.335129 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 02:57:26.335236 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 02:57:26.335365 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":3}
I20260925 02:57:26.336324 26769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.328578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:36001 (local address 127.0.0.1:50696)
0925 02:57:26.328912 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.328958 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:26.329060 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:26.329801 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:26.329807 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:26.329829 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:26.330148 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:26.330161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.331945 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.331950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:26.333842 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:26.333857 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.335571 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.335578 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:26.335776 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:26.335808 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 02:57:26.335867 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 02:57:26.335996 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":1}
I20260925 02:57:26.336689 26341 tablet_service.cc:1467] Processing CreateTablet for tablet d4e15f78cc5b49a6a4cce7a3befb332d (DEFAULT_TABLE table=test-table [id=deb02250406f46c6991564566281092b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:57:26.337106 26341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4e15f78cc5b49a6a4cce7a3befb332d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:26.337209 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.328994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50696 (local address 127.18.196.194:36001)
0925 02:57:26.329459 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:26.329465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:26.329484 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:57:26.329555 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:26.329560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:26.329641 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:26.330291 (+ 650us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:26.330490 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.331736 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.334025 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.335321 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.336654 (+ 1333us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:26.336728 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:26.336785 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 02:57:26.336904 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":1677,"threads_started":1}
I20260925 02:57:26.337950 26481 tablet_service.cc:1467] Processing CreateTablet for tablet d4e15f78cc5b49a6a4cce7a3befb332d (DEFAULT_TABLE table=test-table [id=deb02250406f46c6991564566281092b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:57:26.338346 26481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4e15f78cc5b49a6a4cce7a3befb332d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:26.338419 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.330537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60580 (local address 127.18.196.195:43605)
0925 02:57:26.331015 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:26.331023 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:26.331519 (+ 496us) server_negotiation.cc:407] Connection header received
0925 02:57:26.331685 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:26.331690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:26.331776 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:26.331872 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:26.332837 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.333867 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.336282 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.337480 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.337654 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:26.337721 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:26.337999 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 02:57:26.338137 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":236,"threads_started":1}
I20260925 02:57:26.338419 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.330175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:43605 (local address 127.0.0.1:60580)
0925 02:57:26.331405 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.331421 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:26.331562 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:26.332100 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:26.332104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:26.332130 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:26.332648 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:26.332659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.334030 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.334034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:26.336085 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:26.336101 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.337680 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.337686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:26.337842 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:26.337864 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:57:26.337965 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 02:57:26.338077 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1123,"mutex_wait_us":21,"thread_start_us":995,"threads_started":1}
I20260925 02:57:26.343595 26623 tablet_service.cc:1467] Processing CreateTablet for tablet d4e15f78cc5b49a6a4cce7a3befb332d (DEFAULT_TABLE table=test-table [id=deb02250406f46c6991564566281092b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:57:26.344031 26623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4e15f78cc5b49a6a4cce7a3befb332d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:26.346261 26788 tablet_bootstrap.cc:492] T d4e15f78cc5b49a6a4cce7a3befb332d P bb0d89109f23435db76236a5f64b25d5: Bootstrap starting.
I20260925 02:57:26.347390 26787 tablet_bootstrap.cc:492] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618: Bootstrap starting.
I20260925 02:57:26.348042 26788 tablet_bootstrap.cc:654] T d4e15f78cc5b49a6a4cce7a3befb332d P bb0d89109f23435db76236a5f64b25d5: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:26.348722 26787 tablet_bootstrap.cc:654] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:26.350948 26790 tablet_bootstrap.cc:492] T d4e15f78cc5b49a6a4cce7a3befb332d P 920f7345bb8940ff98552b67ad8fb652: Bootstrap starting.
I20260925 02:57:26.351092 26787 tablet_bootstrap.cc:492] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618: No bootstrap required, opened a new log
I20260925 02:57:26.351212 26787 ts_tablet_manager.cc:1397] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20260925 02:57:26.351794 26788 tablet_bootstrap.cc:492] T d4e15f78cc5b49a6a4cce7a3befb332d P bb0d89109f23435db76236a5f64b25d5: No bootstrap required, opened a new log
I20260925 02:57:26.351939 26788 ts_tablet_manager.cc:1397] T d4e15f78cc5b49a6a4cce7a3befb332d P bb0d89109f23435db76236a5f64b25d5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:57:26.351979 26787 raft_consensus.cc:348] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2fa8d55ea204f48a127e243690cb618" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38247 } } peers { permanent_uuid: "bb0d89109f23435db76236a5f64b25d5" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 36001 } } peers { permanent_uuid: "920f7345bb8940ff98552b67ad8fb652" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43605 } }
I20260925 02:57:26.352218 26787 raft_consensus.cc:374] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:26.352319 26787 raft_consensus.cc:729] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2fa8d55ea204f48a127e243690cb618, State: Initialized, Role: FOLLOWER
I20260925 02:57:26.352572 26790 tablet_bootstrap.cc:654] T d4e15f78cc5b49a6a4cce7a3befb332d P 920f7345bb8940ff98552b67ad8fb652: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:26.352561 26787 consensus_queue.cc:260] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2fa8d55ea204f48a127e243690cb618" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38247 } } peers { permanent_uuid: "bb0d89109f23435db76236a5f64b25d5" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 36001 } } peers { permanent_uuid: "920f7345bb8940ff98552b67ad8fb652" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43605 } }
I20260925 02:57:26.352718 26788 raft_consensus.cc:348] T d4e15f78cc5b49a6a4cce7a3befb332d P bb0d89109f23435db76236a5f64b25d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2fa8d55ea204f48a127e243690cb618" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38247 } } peers { permanent_uuid: "bb0d89109f23435db76236a5f64b25d5" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 36001 } } peers { permanent_uuid: "920f7345bb8940ff98552b67ad8fb652" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43605 } }
I20260925 02:57:26.353127 26788 raft_consensus.cc:374] T d4e15f78cc5b49a6a4cce7a3befb332d P bb0d89109f23435db76236a5f64b25d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:26.353259 26788 raft_consensus.cc:729] T d4e15f78cc5b49a6a4cce7a3befb332d P bb0d89109f23435db76236a5f64b25d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb0d89109f23435db76236a5f64b25d5, State: Initialized, Role: FOLLOWER
I20260925 02:57:26.353469 26787 ts_tablet_manager.cc:1428] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:57:26.353492 26788 consensus_queue.cc:260] T d4e15f78cc5b49a6a4cce7a3befb332d P bb0d89109f23435db76236a5f64b25d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2fa8d55ea204f48a127e243690cb618" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38247 } } peers { permanent_uuid: "bb0d89109f23435db76236a5f64b25d5" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 36001 } } peers { permanent_uuid: "920f7345bb8940ff98552b67ad8fb652" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43605 } }
I20260925 02:57:26.357405 26406 heartbeater.cc:503] Master 127.18.196.252:33819 was elected leader, sending a full tablet report...
I20260925 02:57:26.360194 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.354736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.18.196.193:55025)
0925 02:57:26.359650 (+ 4914us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.359856 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4735,"thread_start_us":4558,"threads_started":1}
I20260925 02:57:26.360884 26790 tablet_bootstrap.cc:492] T d4e15f78cc5b49a6a4cce7a3befb332d P 920f7345bb8940ff98552b67ad8fb652: No bootstrap required, opened a new log
I20260925 02:57:26.360908 26788 ts_tablet_manager.cc:1428] T d4e15f78cc5b49a6a4cce7a3befb332d P bb0d89109f23435db76236a5f64b25d5: Time spent starting tablet: real 0.009s user 0.002s sys 0.007s
I20260925 02:57:26.361065 26790 ts_tablet_manager.cc:1397] T d4e15f78cc5b49a6a4cce7a3befb332d P 920f7345bb8940ff98552b67ad8fb652: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260925 02:57:26.362437 26796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.361436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.18.196.194:48427)
0925 02:57:26.362007 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.362163 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":308,"threads_started":1}
I20260925 02:57:26.362393 26790 raft_consensus.cc:348] T d4e15f78cc5b49a6a4cce7a3befb332d P 920f7345bb8940ff98552b67ad8fb652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2fa8d55ea204f48a127e243690cb618" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38247 } } peers { permanent_uuid: "bb0d89109f23435db76236a5f64b25d5" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 36001 } } peers { permanent_uuid: "920f7345bb8940ff98552b67ad8fb652" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43605 } }
I20260925 02:57:26.362697 26790 raft_consensus.cc:374] T d4e15f78cc5b49a6a4cce7a3befb332d P 920f7345bb8940ff98552b67ad8fb652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:26.362735 26546 heartbeater.cc:503] Master 127.18.196.252:33819 was elected leader, sending a full tablet report...
I20260925 02:57:26.362891 26790 raft_consensus.cc:729] T d4e15f78cc5b49a6a4cce7a3befb332d P 920f7345bb8940ff98552b67ad8fb652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 920f7345bb8940ff98552b67ad8fb652, State: Initialized, Role: FOLLOWER
I20260925 02:57:26.363217 26790 consensus_queue.cc:260] T d4e15f78cc5b49a6a4cce7a3befb332d P 920f7345bb8940ff98552b67ad8fb652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2fa8d55ea204f48a127e243690cb618" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38247 } } peers { permanent_uuid: "bb0d89109f23435db76236a5f64b25d5" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 36001 } } peers { permanent_uuid: "920f7345bb8940ff98552b67ad8fb652" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43605 } }
I20260925 02:57:26.363734 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.363289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.18.196.193:38487)
0925 02:57:26.363456 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.363568 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 02:57:26.365821 26790 ts_tablet_manager.cc:1428] T d4e15f78cc5b49a6a4cce7a3befb332d P 920f7345bb8940ff98552b67ad8fb652: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260925 02:57:26.366353 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.363089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.18.196.194:39983)
0925 02:57:26.365142 (+ 2053us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.366077 (+ 935us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1926,"thread_start_us":1801,"threads_started":1}
I20260925 02:57:26.367300 26799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.366437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.18.196.195:55741)
0925 02:57:26.366939 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.367065 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"threads_started":1}
I20260925 02:57:26.368003 26688 heartbeater.cc:503] Master 127.18.196.252:33819 was elected leader, sending a full tablet report...
I20260925 02:57:26.368455 26799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.368048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.18.196.195:55071)
0925 02:57:26.368239 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.368316 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260925 02:57:26.393338 26792 raft_consensus.cc:482] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:57:26.393523 26792 raft_consensus.cc:504] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2fa8d55ea204f48a127e243690cb618" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38247 } } peers { permanent_uuid: "bb0d89109f23435db76236a5f64b25d5" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 36001 } } peers { permanent_uuid: "920f7345bb8940ff98552b67ad8fb652" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43605 } }
I20260925 02:57:26.394181 26792 leader_election.cc:288] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb0d89109f23435db76236a5f64b25d5 (127.18.196.194:36001), 920f7345bb8940ff98552b67ad8fb652 (127.18.196.195:43605)
I20260925 02:57:26.394629 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4e15f78cc5b49a6a4cce7a3befb332d" candidate_uuid: "c2fa8d55ea204f48a127e243690cb618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb0d89109f23435db76236a5f64b25d5" is_pre_election: true
I20260925 02:57:26.394629 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4e15f78cc5b49a6a4cce7a3befb332d" candidate_uuid: "c2fa8d55ea204f48a127e243690cb618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "920f7345bb8940ff98552b67ad8fb652" is_pre_election: true
I20260925 02:57:26.394893 26643 raft_consensus.cc:2454] T d4e15f78cc5b49a6a4cce7a3befb332d P 920f7345bb8940ff98552b67ad8fb652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2fa8d55ea204f48a127e243690cb618 in term 0.
I20260925 02:57:26.394893 26501 raft_consensus.cc:2454] T d4e15f78cc5b49a6a4cce7a3befb332d P bb0d89109f23435db76236a5f64b25d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2fa8d55ea204f48a127e243690cb618 in term 0.
I20260925 02:57:26.395385 26293 leader_election.cc:302] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 920f7345bb8940ff98552b67ad8fb652, c2fa8d55ea204f48a127e243690cb618; no voters:
I20260925 02:57:26.395675 26792 raft_consensus.cc:2781] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:57:26.395803 26792 raft_consensus.cc:482] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:57:26.395913 26792 raft_consensus.cc:3037] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:26.398025 26792 raft_consensus.cc:504] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2fa8d55ea204f48a127e243690cb618" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38247 } } peers { permanent_uuid: "bb0d89109f23435db76236a5f64b25d5" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 36001 } } peers { permanent_uuid: "920f7345bb8940ff98552b67ad8fb652" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43605 } }
I20260925 02:57:26.398605 26792 leader_election.cc:288] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618 [CANDIDATE]: Term 1 election: Requested vote from peers bb0d89109f23435db76236a5f64b25d5 (127.18.196.194:36001), 920f7345bb8940ff98552b67ad8fb652 (127.18.196.195:43605)
I20260925 02:57:26.399078 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4e15f78cc5b49a6a4cce7a3befb332d" candidate_uuid: "c2fa8d55ea204f48a127e243690cb618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "920f7345bb8940ff98552b67ad8fb652"
I20260925 02:57:26.399115 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4e15f78cc5b49a6a4cce7a3befb332d" candidate_uuid: "c2fa8d55ea204f48a127e243690cb618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb0d89109f23435db76236a5f64b25d5"
I20260925 02:57:26.399269 26643 raft_consensus.cc:3037] T d4e15f78cc5b49a6a4cce7a3befb332d P 920f7345bb8940ff98552b67ad8fb652 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:26.399309 26501 raft_consensus.cc:3037] T d4e15f78cc5b49a6a4cce7a3befb332d P bb0d89109f23435db76236a5f64b25d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:26.401227 26501 raft_consensus.cc:2454] T d4e15f78cc5b49a6a4cce7a3befb332d P bb0d89109f23435db76236a5f64b25d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2fa8d55ea204f48a127e243690cb618 in term 1.
I20260925 02:57:26.401387 26643 raft_consensus.cc:2454] T d4e15f78cc5b49a6a4cce7a3befb332d P 920f7345bb8940ff98552b67ad8fb652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2fa8d55ea204f48a127e243690cb618 in term 1.
I20260925 02:57:26.401636 26295 leader_election.cc:302] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb0d89109f23435db76236a5f64b25d5, c2fa8d55ea204f48a127e243690cb618; no voters:
I20260925 02:57:26.401928 26792 raft_consensus.cc:2781] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:26.402313 26792 raft_consensus.cc:686] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618 [term 1 LEADER]: Becoming Leader. State: Replica: c2fa8d55ea204f48a127e243690cb618, State: Running, Role: LEADER
I20260925 02:57:26.402663 26792 consensus_queue.cc:237] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2fa8d55ea204f48a127e243690cb618" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38247 } } peers { permanent_uuid: "bb0d89109f23435db76236a5f64b25d5" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 36001 } } peers { permanent_uuid: "920f7345bb8940ff98552b67ad8fb652" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43605 } }
I20260925 02:57:26.403378 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.402991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.18.196.193:50387)
0925 02:57:26.403127 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.403227 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 02:57:26.405315 26224 catalog_manager.cc:5495] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2fa8d55ea204f48a127e243690cb618 (127.18.196.193). New cstate: current_term: 1 leader_uuid: "c2fa8d55ea204f48a127e243690cb618" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2fa8d55ea204f48a127e243690cb618" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 38247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb0d89109f23435db76236a5f64b25d5" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 36001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "920f7345bb8940ff98552b67ad8fb652" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43605 } health_report { overall_health: UNKNOWN } } }
I20260925 02:57:26.434427 26783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.429595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.18.196.193:38247)
0925 02:57:26.429765 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:26.429771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:26.429866 (+ 95us) server_negotiation.cc:407] Connection header received
0925 02:57:26.430104 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:26.430108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:26.430169 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:26.430265 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:57:26.430665 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.431377 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.432727 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.433159 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.433293 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:26.433704 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:57:26.433808 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:57:26.433924 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:26.434094 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 02:57:26.434169 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:57:26.437454 26643 raft_consensus.cc:1261] T d4e15f78cc5b49a6a4cce7a3befb332d P 920f7345bb8940ff98552b67ad8fb652 [term 1 FOLLOWER]: Refusing update from remote peer c2fa8d55ea204f48a127e243690cb618: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:57:26.437484 26501 raft_consensus.cc:1261] T d4e15f78cc5b49a6a4cce7a3befb332d P bb0d89109f23435db76236a5f64b25d5 [term 1 FOLLOWER]: Refusing update from remote peer c2fa8d55ea204f48a127e243690cb618: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:57:26.438083 26792 consensus_queue.cc:1035] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618 [LEADER]: Connected to new peer: Peer: permanent_uuid: "920f7345bb8940ff98552b67ad8fb652" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 43605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:57:26.438366 26800 consensus_queue.cc:1035] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb0d89109f23435db76236a5f64b25d5" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:57:26.440963 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.440426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.18.196.194:50443)
0925 02:57:26.440644 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.440731 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260925 02:57:26.441998 26799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.440407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.18.196.195:56349)
0925 02:57:26.441707 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.441802 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1172}
I20260925 02:57:26.446951 26799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.446445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.18.196.195:38605)
0925 02:57:26.446656 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.446741 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260925 02:57:26.446949 26804 mvcc.cc:205] Tried to move back new op lower bound from 7333089470201819136 to 7333089470067453952. Current Snapshot: MvccSnapshot[applied={T|T < 7333089470201819136}]
I20260925 02:57:26.447186 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.446444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.18.196.193:47355)
0925 02:57:26.446922 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.447024 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348}
I20260925 02:57:26.451154 26805 mvcc.cc:205] Tried to move back new op lower bound from 7333089470201819136 to 7333089470067453952. Current Snapshot: MvccSnapshot[applied={T|T < 7333089470201819136}]
I20260925 02:57:26.451325 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.450279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.18.196.193:57621)
0925 02:57:26.451077 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.451169 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":693}
I20260925 02:57:26.453042 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.452633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.18.196.194:48799)
0925 02:57:26.452769 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.452877 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 02:57:26.457576 26225 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:57:26.457831 26225 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:57:26.462915 26225 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 469b93b3d2084e29807a456e82a08b40: Sending DeleteTablet for 3 replicas of tablet d4e15f78cc5b49a6a4cce7a3befb332d
I20260925 02:57:26.464028 26341 tablet_service.cc:1514] Processing DeleteTablet for tablet d4e15f78cc5b49a6a4cce7a3befb332d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:26 UTC) from {username='oryx', principal='oryx/127.18.196.252@KRBTEST.COM'} at 127.0.0.1:46320
I20260925 02:57:26.464226 26481 tablet_service.cc:1514] Processing DeleteTablet for tablet d4e15f78cc5b49a6a4cce7a3befb332d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:26 UTC) from {username='oryx', principal='oryx/127.18.196.252@KRBTEST.COM'} at 127.0.0.1:50696
I20260925 02:57:26.464344 26623 tablet_service.cc:1514] Processing DeleteTablet for tablet d4e15f78cc5b49a6a4cce7a3befb332d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:26 UTC) from {username='oryx', principal='oryx/127.18.196.252@KRBTEST.COM'} at 127.0.0.1:60580
I20260925 02:57:26.465009 26812 tablet_replica.cc:330] T d4e15f78cc5b49a6a4cce7a3befb332d P bb0d89109f23435db76236a5f64b25d5: stopping tablet replica
I20260925 02:57:26.465493 26812 raft_consensus.cc:2229] T d4e15f78cc5b49a6a4cce7a3befb332d P bb0d89109f23435db76236a5f64b25d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:57:26.465934 26812 raft_consensus.cc:2258] T d4e15f78cc5b49a6a4cce7a3befb332d P bb0d89109f23435db76236a5f64b25d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:57:26.466118 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 26200
I20260925 02:57:26.469470 26812 ts_tablet_manager.cc:1905] T d4e15f78cc5b49a6a4cce7a3befb332d P bb0d89109f23435db76236a5f64b25d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:57:26.478310 26812 ts_tablet_manager.cc:1918] T d4e15f78cc5b49a6a4cce7a3befb332d P bb0d89109f23435db76236a5f64b25d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:57:26.479053 26813 tablet_replica.cc:330] T d4e15f78cc5b49a6a4cce7a3befb332d P 920f7345bb8940ff98552b67ad8fb652: stopping tablet replica
I20260925 02:57:26.479276 26812 log.cc:1196] T d4e15f78cc5b49a6a4cce7a3befb332d P bb0d89109f23435db76236a5f64b25d5: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/d4e15f78cc5b49a6a4cce7a3befb332d
I20260925 02:57:26.479527 26813 raft_consensus.cc:2229] T d4e15f78cc5b49a6a4cce7a3befb332d P 920f7345bb8940ff98552b67ad8fb652 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:57:26.479703 26811 tablet_replica.cc:330] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618: stopping tablet replica
I20260925 02:57:26.480007 26811 raft_consensus.cc:2229] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:57:26.480113 26812 ts_tablet_manager.cc:1939] T d4e15f78cc5b49a6a4cce7a3befb332d P bb0d89109f23435db76236a5f64b25d5: Deleting consensus metadata
I20260925 02:57:26.480466 26811 raft_consensus.cc:2258] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:57:26.480896 26813 raft_consensus.cc:2258] T d4e15f78cc5b49a6a4cce7a3befb332d P 920f7345bb8940ff98552b67ad8fb652 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:57:26.481261 26212 catalog_manager.cc:4841] TS bb0d89109f23435db76236a5f64b25d5 (127.18.196.194:36001): tablet d4e15f78cc5b49a6a4cce7a3befb332d (table test-table [id=deb02250406f46c6991564566281092b]) successfully deleted
I20260925 02:57:26.481877 26811 ts_tablet_manager.cc:1905] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:57:26.483688 26813 ts_tablet_manager.cc:1905] T d4e15f78cc5b49a6a4cce7a3befb332d P 920f7345bb8940ff98552b67ad8fb652: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:57:26.488099 26811 ts_tablet_manager.cc:1918] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:57:26.488268 26811 log.cc:1196] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-0/wal/wals/d4e15f78cc5b49a6a4cce7a3befb332d
I20260925 02:57:26.488791 26811 ts_tablet_manager.cc:1939] T d4e15f78cc5b49a6a4cce7a3befb332d P c2fa8d55ea204f48a127e243690cb618: Deleting consensus metadata
I20260925 02:57:26.489933 26209 catalog_manager.cc:4841] TS c2fa8d55ea204f48a127e243690cb618 (127.18.196.193:38247): tablet d4e15f78cc5b49a6a4cce7a3befb332d (table test-table [id=deb02250406f46c6991564566281092b]) successfully deleted
I20260925 02:57:26.490171 26813 ts_tablet_manager.cc:1918] T d4e15f78cc5b49a6a4cce7a3befb332d P 920f7345bb8940ff98552b67ad8fb652: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:57:26.490284 26813 log.cc:1196] T d4e15f78cc5b49a6a4cce7a3befb332d P 920f7345bb8940ff98552b67ad8fb652: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/d4e15f78cc5b49a6a4cce7a3befb332d
I20260925 02:57:26.490597 26813 ts_tablet_manager.cc:1939] T d4e15f78cc5b49a6a4cce7a3befb332d P 920f7345bb8940ff98552b67ad8fb652: Deleting consensus metadata
I20260925 02:57:26.491515 26210 catalog_manager.cc:4841] TS 920f7345bb8940ff98552b67ad8fb652 (127.18.196.195:43605): tablet d4e15f78cc5b49a6a4cce7a3befb332d (table test-table [id=deb02250406f46c6991564566281092b]) successfully deleted
I20260925 02:57:26.494407 26783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.467050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46340 (local address 127.18.196.193:38247)
0925 02:57:26.467234 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:26.467240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:26.479298 (+ 12058us) server_negotiation.cc:407] Connection header received
0925 02:57:26.479508 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:26.479514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:26.479599 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:26.479718 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:26.481640 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.482436 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.483955 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.484387 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.484530 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:26.487358 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:26.487388 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:26.487405 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:26.487471 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:26.491134 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:26.492238 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:26.492248 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:26.492251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:26.492410 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:26.492823 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:26.492828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:26.492832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:26.493218 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:26.493521 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:26.493841 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 02:57:26.494071 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:57:26.495079 26401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:26.616047 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 26200
I20260925 02:57:26.639353 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 26418
I20260925 02:57:26.652640 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.640123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50700 (local address 127.18.196.194:36001)
0925 02:57:26.640370 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:26.640375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:26.640414 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:57:26.640588 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:26.640593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:26.640665 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:26.640746 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:26.642034 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.642704 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.643996 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.644439 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.644537 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:26.647004 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:26.647020 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:26.647037 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:26.647070 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:26.649949 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:26.650567 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:26.650574 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:26.650576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:26.650698 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:26.651216 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:26.651220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:26.651223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:26.651636 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:26.651866 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:26.652133 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 02:57:26.652294 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260925 02:57:26.653378 26541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:26.723970 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.723562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.0.0.1:43898)
0925 02:57:26.723717 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.723811 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 02:57:26.724948 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.724650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40775 (local address 127.0.0.1:43914)
0925 02:57:26.724769 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:26.724830 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 02:57:26.746661 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 26418
I20260925 02:57:26.765103 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 26559
I20260925 02:57:26.778041 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:26.765860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60594 (local address 127.18.196.195:43605)
0925 02:57:26.766051 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:26.766056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:26.766123 (+ 67us) server_negotiation.cc:407] Connection header received
0925 02:57:26.766301 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:26.766306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:26.766373 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:26.766473 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:26.767850 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.768551 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.769730 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:26.770117 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:26.770208 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:26.772659 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:26.772675 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:26.772691 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:26.772727 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:26.775777 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:26.776359 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:26.776366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:26.776368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:26.776501 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:26.776807 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:26.776810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:26.776812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:26.777165 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:26.777370 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:26.777626 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 02:57:26.777788 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:57:26.778684 26683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:26.874014 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 26559
I20260925 02:57:26.892117 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 25892
I20260925 02:57:26.893162 26065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:27.010746 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 25892
I20260925 02:57:27.031985 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 25993
I20260925 02:57:27.033078 26255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:27.156117 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 25993
2026-09-25T02:57:27Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9314 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:57:27 dist-test-slave-8zvk krb5kdc[26823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:57:27 dist-test-slave-8zvk krb5kdc[26823](info): set up 2 sockets
Sep 25 02:57:27 dist-test-slave-8zvk krb5kdc[26823](info): commencing operation
krb5kdc: starting...
W20260925 02:57:28.279277 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 02:57:28 dist-test-slave-8zvk krb5kdc[26823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:57:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:57:28Z Disabled control of system clock
WARNING: no policy specified for oryx/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.254.keytab.
Entry for principal oryx/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.254.keytab.
I20260925 02:57:28.486071 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:34109
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.254.keytab
--principal=oryx/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:45625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:34109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:28.797871 26836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:28.798144 26836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:28.798300 26836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:28.804773 26836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:28.804852 26836 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 02:57:28.804929 26836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:28.805020 26836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:28.805102 26836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:28.805220 26836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:28.815551 26836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:45625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:34109
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.254.keytab
--principal=oryx/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:34109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:28.816756 26836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:28.818001 26836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:28.828914 26841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:28.828948 26844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:28.828994 26842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:28.829897 26836 server_base.cc:1034] running on GCE node
I20260925 02:57:28.830588 26836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:28.831895 26836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:28.833127 26836 hybrid_clock.cc:648] HybridClock initialized: now 1790305048833072 us; error 34 us; skew 500 ppm
I20260925 02:57:28.833441 26836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:28 dist-test-slave-8zvk krb5kdc[26823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305048, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:28.836295 26836 init.cc:377] Logged in from keytab as oryx/127.18.196.254@KRBTEST.COM (short username oryx)
I20260925 02:57:28.837558 26836 webserver.cc:460] Webserver started at http://127.18.196.254:43271/ using document root <none> and password file <none>
I20260925 02:57:28.838188 26836 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:28.838310 26836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:28.838619 26836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:28.841348 26836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "eed677241c274560b0d85c7eabd90e0a"
format_stamp: "Formatted at 2026-09-25 02:57:28 on dist-test-slave-8zvk"
server_key: "c27cf3fa556be197277f9409d1e6e8d6"
server_key_iv: "25c7b3fe3441f2c57ce59673d43e3e35"
server_key_version: "encryptionkey@0"
I20260925 02:57:28.842163 26836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "eed677241c274560b0d85c7eabd90e0a"
format_stamp: "Formatted at 2026-09-25 02:57:28 on dist-test-slave-8zvk"
server_key: "c27cf3fa556be197277f9409d1e6e8d6"
server_key_iv: "25c7b3fe3441f2c57ce59673d43e3e35"
server_key_version: "encryptionkey@0"
I20260925 02:57:28.846786 26836 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 02:57:28.849814 26851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:28.851034 26836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:57:28.851207 26836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "eed677241c274560b0d85c7eabd90e0a"
format_stamp: "Formatted at 2026-09-25 02:57:28 on dist-test-slave-8zvk"
server_key: "c27cf3fa556be197277f9409d1e6e8d6"
server_key_iv: "25c7b3fe3441f2c57ce59673d43e3e35"
server_key_version: "encryptionkey@0"
I20260925 02:57:28.851408 26836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:28.863828 26836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:28.867359 26836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:28.867619 26836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:28.887720 26836 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:34109
I20260925 02:57:28.887750 26912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:34109 every 8 connection(s)
I20260925 02:57:28.888919 26836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:57:28.892503 26913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:28.894266 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 26836
I20260925 02:57:28.894464 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:57:28.895206 19219 external_mini_cluster.cc:1428] Setting key e856d9d07f41cbbd0d55be23fbccc2fc
I20260925 02:57:28.905211 26913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a: Bootstrap starting.
I20260925 02:57:28.908607 26913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:28.909622 26913 log.cc:824] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:57:28 dist-test-slave-8zvk krb5kdc[26823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM
I20260925 02:57:28.912321 26913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a: No bootstrap required, opened a new log
I20260925 02:57:28.918171 26916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:28.897109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42302 (local address 127.18.196.254:34109)
0925 02:57:28.897952 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:28.897973 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:28.898027 (+ 54us) server_negotiation.cc:407] Connection header received
0925 02:57:28.900342 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:28.900366 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:28.900806 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:28.901188 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:28.902484 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:28.904527 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:28.905799 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:28.906877 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:28.907070 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:28.911086 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:28.911093 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:28.911105 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:28.911140 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:28.914110 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:28.914951 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:28.914956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:28.914961 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:28.915144 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:28.915562 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:28.915566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:28.915568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:28.916088 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:28.916299 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:28.916819 (+ 520us) server_negotiation.cc:299] Negotiation successful
0925 02:57:28.917232 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":338,"threads_started":1}
I20260925 02:57:28.920590 26913 raft_consensus.cc:348] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed677241c274560b0d85c7eabd90e0a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 34109 } }
I20260925 02:57:28.920917 26913 raft_consensus.cc:374] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:28.921017 26913 raft_consensus.cc:729] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed677241c274560b0d85c7eabd90e0a, State: Initialized, Role: FOLLOWER
I20260925 02:57:28.921569 26913 consensus_queue.cc:260] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed677241c274560b0d85c7eabd90e0a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 34109 } }
I20260925 02:57:28.921808 26913 raft_consensus.cc:388] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:57:28.921959 26913 raft_consensus.cc:482] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:57:28.922133 26913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:28.924187 26913 raft_consensus.cc:504] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed677241c274560b0d85c7eabd90e0a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 34109 } }
I20260925 02:57:28.924738 26913 leader_election.cc:302] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eed677241c274560b0d85c7eabd90e0a; no voters:
I20260925 02:57:28.925275 26913 leader_election.cc:288] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:57:28.925397 26918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:28.926102 26918 raft_consensus.cc:686] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a [term 1 LEADER]: Becoming Leader. State: Replica: eed677241c274560b0d85c7eabd90e0a, State: Running, Role: LEADER
I20260925 02:57:28.926626 26918 consensus_queue.cc:237] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed677241c274560b0d85c7eabd90e0a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 34109 } }
I20260925 02:57:28.927276 26913 sys_catalog.cc:564] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:28.929134 26919 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eed677241c274560b0d85c7eabd90e0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed677241c274560b0d85c7eabd90e0a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 34109 } } }
I20260925 02:57:28.929484 26919 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:28.929757 26920 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader eed677241c274560b0d85c7eabd90e0a. Latest consensus state: current_term: 1 leader_uuid: "eed677241c274560b0d85c7eabd90e0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed677241c274560b0d85c7eabd90e0a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 34109 } } }
I20260925 02:57:28.930384 26920 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:28.931661 26923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:28.936347 26923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:28.945361 26923 catalog_manager.cc:1347] Generated new cluster ID: 9184d1c6692f42ec9013362f4c0f5362
I20260925 02:57:28.945472 26923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:28.980654 26923 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:57:28.982041 26923 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:28.995697 26923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a: Generated new TSK 0
I20260925 02:57:28.996450 26923 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.18.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.193.keytab.
Entry for principal oryx/127.18.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.193.keytab.
I20260925 02:57:29.054059 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:34109
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.193.keytab
--principal=oryx/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:45625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:29.365947 26939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:29.366245 26939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:29.366412 26939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:29.373452 26939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:29.373611 26939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:29.373751 26939 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:57:29.373879 26939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 02:57:29.383435 26939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:45625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.193.keytab
--principal=oryx/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.196.254:34109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:29.384656 26939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:29.385939 26939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:29.399084 26944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:29.399125 26947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:29.399071 26945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:29.399657 26939 server_base.cc:1034] running on GCE node
I20260925 02:57:29.400286 26939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:29.401162 26939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:29.402412 26939 hybrid_clock.cc:648] HybridClock initialized: now 1790305049402334 us; error 36 us; skew 500 ppm
I20260925 02:57:29.402704 26939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:29 dist-test-slave-8zvk krb5kdc[26823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305049, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:29.405728 26939 init.cc:377] Logged in from keytab as oryx/127.18.196.193@KRBTEST.COM (short username oryx)
I20260925 02:57:29.407132 26939 webserver.cc:460] Webserver started at http://127.18.196.193:44271/ using document root <none> and password file <none>
I20260925 02:57:29.407766 26939 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:29.407846 26939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:29.408100 26939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:29.410894 26939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "45a8bfe724a24cadb6ec77a4a749b7ef"
format_stamp: "Formatted at 2026-09-25 02:57:29 on dist-test-slave-8zvk"
server_key: "a46ec8492d10d0a01f23b73cb6c7ddc6"
server_key_iv: "17bd5153c9b2d92dc02d3be48c0b55ff"
server_key_version: "encryptionkey@0"
I20260925 02:57:29.411677 26939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "45a8bfe724a24cadb6ec77a4a749b7ef"
format_stamp: "Formatted at 2026-09-25 02:57:29 on dist-test-slave-8zvk"
server_key: "a46ec8492d10d0a01f23b73cb6c7ddc6"
server_key_iv: "17bd5153c9b2d92dc02d3be48c0b55ff"
server_key_version: "encryptionkey@0"
I20260925 02:57:29.416437 26939 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260925 02:57:29.419996 26954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:29.421298 26939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:57:29.421509 26939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "45a8bfe724a24cadb6ec77a4a749b7ef"
format_stamp: "Formatted at 2026-09-25 02:57:29 on dist-test-slave-8zvk"
server_key: "a46ec8492d10d0a01f23b73cb6c7ddc6"
server_key_iv: "17bd5153c9b2d92dc02d3be48c0b55ff"
server_key_version: "encryptionkey@0"
I20260925 02:57:29.421728 26939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:29.433002 26939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:29.436767 26939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:29.437031 26939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:29.439576 26939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:29.439677 26939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:29.439810 26939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:29.439908 26939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:57:29 dist-test-slave-8zvk krb5kdc[26823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305049, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM
I20260925 02:57:29.462106 26963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:29.440163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:47757 (local address 127.18.196.254:34109)
0925 02:57:29.442469 (+ 2306us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:29.442477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:29.442503 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:57:29.444152 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:29.444157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:29.444245 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:29.444368 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:29.446475 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:29.447773 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:29.450225 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:29.450668 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:29.450800 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:29.455548 (+ 4748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:29.455556 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:29.455560 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:29.455607 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:29.458809 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:29.459725 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:29.459731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:29.459734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:29.459877 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:29.460415 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:29.460423 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:29.460427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:29.460700 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:29.460848 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:29.461517 (+ 669us) server_negotiation.cc:299] Negotiation successful
0925 02:57:29.461724 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2174,"thread_start_us":2058,"threads_started":1}
I20260925 02:57:29.463296 26964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:29.440714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:34109 (local address 127.18.196.193:47757)
0925 02:57:29.441774 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:29.441829 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:29.443860 (+ 2031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:29.444612 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:29.444619 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:29.445370 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:29.446241 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:29.446269 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:29.447935 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:29.447943 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:29.450069 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:29.450081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:29.450940 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:29.450947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:29.451198 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:29.452031 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:29.452063 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:29.455222 (+ 3159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:29.458960 (+ 3738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:29.458974 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:29.458988 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:29.459610 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:29.460034 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:29.460041 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:29.460046 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:29.460296 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:29.460825 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:29.460832 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:29.461166 (+ 334us) client_negotiation.cc:769] Sending connection context
0925 02:57:29.461437 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 02:57:29.461916 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":431,"threads_started":1}
I20260925 02:57:29.484884 26963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:29.469300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:41971 (local address 127.18.196.254:34109)
0925 02:57:29.469519 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:29.469528 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:29.470082 (+ 554us) server_negotiation.cc:407] Connection header received
0925 02:57:29.470262 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:29.470265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:29.470321 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:29.470395 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:29.471952 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:29.472938 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:29.475573 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:29.476360 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:29.476500 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:29.479094 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:29.479100 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:29.479105 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:29.479142 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:29.482070 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:29.482992 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:29.482997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:29.483001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:29.483123 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:29.483480 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:29.483485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:29.483488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:29.483752 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:29.483896 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:29.484319 (+ 423us) server_negotiation.cc:299] Negotiation successful
0925 02:57:29.484527 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:57:29.484983 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:29.469247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:34109 (local address 127.18.196.193:41971)
0925 02:57:29.469962 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:29.469987 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:29.470143 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:29.470460 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:29.470467 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:29.471198 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:29.471717 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:29.471732 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:29.473056 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:29.473065 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:29.475359 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:29.475376 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:29.476608 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:29.476618 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:29.476829 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:29.477480 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:29.477509 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:29.478850 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:29.482206 (+ 3356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:29.482211 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:29.482215 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:29.482876 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:29.483240 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:29.483244 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:29.483248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:29.483368 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:29.483877 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:29.483882 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:29.484059 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 02:57:29.484208 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 02:57:29.484524 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":360,"threads_started":1}
I20260925 02:57:29.494733 26939 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:33577
I20260925 02:57:29.494817 27076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:33577 every 8 connection(s)
I20260925 02:57:29.496060 26939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 02:57:29.498114 27077 heartbeater.cc:347] Connected to a master server at 127.18.196.254:34109
I20260925 02:57:29.498296 27077 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:29.498922 27077 heartbeater.cc:511] Master 127.18.196.254:34109 requested a full tablet report, sending...
I20260925 02:57:29.500758 26868 ts_manager.cc:194] Registered new tserver with Master: 45a8bfe724a24cadb6ec77a4a749b7ef (127.18.196.193:33577)
I20260925 02:57:29.502205 26868 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.196.193@KRBTEST.COM'} at 127.18.196.193:47757
I20260925 02:57:29.503145 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 26939
I20260925 02:57:29.503336 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 02:57:29.503953 19219 external_mini_cluster.cc:1428] Setting key 8e44e263073afa8a35099d169cedf7ec
WARNING: no policy specified for oryx/127.18.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.194.keytab.
Entry for principal oryx/127.18.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.194.keytab.
I20260925 02:57:29.549496 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:34109
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.194.keytab
--principal=oryx/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:45625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:29.865923 27081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:29.866220 27081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:29.866333 27081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:29.873188 27081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:29.873286 27081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:29.873363 27081 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:57:29.873440 27081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 02:57:29.882898 27081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:45625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.194.keytab
--principal=oryx/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.196.254:34109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:29.884042 27081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:29.885278 27081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:29.900883 27087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:29.900971 27089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:29.901114 27086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:29.901822 27081 server_base.cc:1034] running on GCE node
I20260925 02:57:29.902484 27081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:29.903486 27081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:29.904729 27081 hybrid_clock.cc:648] HybridClock initialized: now 1790305049904654 us; error 51 us; skew 500 ppm
I20260925 02:57:29.905014 27081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:29 dist-test-slave-8zvk krb5kdc[26823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305049, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:29.907980 27081 init.cc:377] Logged in from keytab as oryx/127.18.196.194@KRBTEST.COM (short username oryx)
I20260925 02:57:29.909235 27081 webserver.cc:460] Webserver started at http://127.18.196.194:42171/ using document root <none> and password file <none>
I20260925 02:57:29.909909 27081 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:29.910036 27081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:29.910331 27081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:29.913148 27081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "fd65e5165204496787f7e3023d03afce"
format_stamp: "Formatted at 2026-09-25 02:57:29 on dist-test-slave-8zvk"
server_key: "cb0b0ac282d80e0092d8cddf4aae8d3b"
server_key_iv: "1c30d18e653bb45235ed7adda068255a"
server_key_version: "encryptionkey@0"
I20260925 02:57:29.913913 27081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "fd65e5165204496787f7e3023d03afce"
format_stamp: "Formatted at 2026-09-25 02:57:29 on dist-test-slave-8zvk"
server_key: "cb0b0ac282d80e0092d8cddf4aae8d3b"
server_key_iv: "1c30d18e653bb45235ed7adda068255a"
server_key_version: "encryptionkey@0"
I20260925 02:57:29.918654 27081 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 02:57:29.921926 27096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:29.923183 27081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:57:29.923403 27081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "fd65e5165204496787f7e3023d03afce"
format_stamp: "Formatted at 2026-09-25 02:57:29 on dist-test-slave-8zvk"
server_key: "cb0b0ac282d80e0092d8cddf4aae8d3b"
server_key_iv: "1c30d18e653bb45235ed7adda068255a"
server_key_version: "encryptionkey@0"
I20260925 02:57:29.923635 27081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:29.949424 27081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:29.953402 27081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:29.953675 27081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:29.956423 27081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:29.956537 27081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:29.956686 27081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:29.956766 27081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:57:29 dist-test-slave-8zvk krb5kdc[26823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305049, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM
I20260925 02:57:29.980240 26963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:29.956758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:35503 (local address 127.18.196.254:34109)
0925 02:57:29.956950 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:29.956957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:29.958484 (+ 1527us) server_negotiation.cc:407] Connection header received
0925 02:57:29.960780 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:29.960785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:29.960884 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:29.960998 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:29.963154 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:29.964295 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:29.967151 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:29.967914 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:29.968072 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:29.972996 (+ 4924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:29.973001 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:29.973005 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:29.973035 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:29.976645 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:29.977630 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:29.977635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:29.977639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:29.977767 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:29.978236 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:29.978245 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:29.978248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:29.978517 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:29.978697 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:29.979644 (+ 947us) server_negotiation.cc:299] Negotiation successful
0925 02:57:29.979866 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 02:57:29.981667 27105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:29.957446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:34109 (local address 127.18.196.194:35503)
0925 02:57:29.958291 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:29.958344 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:29.960467 (+ 2123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:29.961237 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:29.961245 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:29.961947 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:29.962974 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:29.963002 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:29.964497 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:29.964502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:29.966972 (+ 2470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:29.966991 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:29.968206 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:29.968219 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:29.968435 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:29.969172 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:29.969194 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:29.972730 (+ 3536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:29.976798 (+ 4068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:29.976811 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:29.976829 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:29.977460 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:29.977885 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:29.977891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:29.977895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:29.978103 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:29.978664 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:29.978672 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:29.979185 (+ 513us) client_negotiation.cc:769] Sending connection context
0925 02:57:29.979549 (+ 364us) client_negotiation.cc:240] Negotiation successful
0925 02:57:29.980107 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":372,"threads_started":1}
I20260925 02:57:29.999818 27081 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:41945
I20260925 02:57:29.999838 27216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:41945 every 8 connection(s)
I20260925 02:57:30.001153 27081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 02:57:30.003461 27217 heartbeater.cc:347] Connected to a master server at 127.18.196.254:34109
I20260925 02:57:30.003720 27217 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:30.004537 27217 heartbeater.cc:511] Master 127.18.196.254:34109 requested a full tablet report, sending...
I20260925 02:57:30.006001 26868 ts_manager.cc:194] Registered new tserver with Master: fd65e5165204496787f7e3023d03afce (127.18.196.194:41945)
I20260925 02:57:30.006034 26963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:29.989232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:42643 (local address 127.18.196.254:34109)
0925 02:57:29.989418 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:29.989426 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:29.991517 (+ 2091us) server_negotiation.cc:407] Connection header received
0925 02:57:29.991823 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:29.991829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:29.991907 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:29.992008 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:29.993387 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:29.994488 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:29.996543 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:29.997066 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:29.997201 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:29.999899 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:29.999907 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:29.999913 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:29.999956 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:30.003118 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:30.003908 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:30.003914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:30.003917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:30.004071 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:30.004446 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:30.004450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:30.004453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:30.004890 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:30.005030 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:30.005439 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 02:57:30.005647 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:57:30.006079 27193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:29.989224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:34109 (local address 127.18.196.194:42643)
0925 02:57:29.991380 (+ 2156us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:29.991404 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:29.991650 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:29.992041 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:29.992046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:29.992748 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:29.993184 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:29.993202 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:29.994681 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:29.994688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:29.996399 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:29.996415 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:29.997317 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:29.997326 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:29.997619 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:29.998309 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:29.998341 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:29.999700 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:30.003265 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:30.003273 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:30.003279 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:30.003764 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:30.004191 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:30.004195 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:30.004197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:30.004307 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:30.005017 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:30.005021 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:30.005188 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 02:57:30.005324 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 02:57:30.005635 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1988,"thread_start_us":1827,"threads_started":1}
I20260925 02:57:30.007094 26868 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.196.194@KRBTEST.COM'} at 127.18.196.194:35503
I20260925 02:57:30.009272 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 27081
I20260925 02:57:30.009441 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 02:57:30.009948 19219 external_mini_cluster.cc:1428] Setting key e12120e8a8f2242ab8f2e7f56084a711
WARNING: no policy specified for oryx/127.18.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.195.keytab.
Entry for principal oryx/127.18.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.195.keytab.
I20260925 02:57:30.054276 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:34109
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.195.keytab
--principal=oryx/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:45625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:30.361992 27221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:30.362270 27221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:30.362428 27221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:30.371213 27221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:30.371323 27221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:30.371484 27221 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:57:30.371616 27221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 02:57:30.381764 27221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:45625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.195.keytab
--principal=oryx/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.196.254:34109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:30.383129 27221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:30.384634 27221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:30.397715 27226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:30.397676 27227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:30.397756 27229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:30.398715 27221 server_base.cc:1034] running on GCE node
I20260925 02:57:30.399399 27221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:30.400240 27221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:30.401484 27221 hybrid_clock.cc:648] HybridClock initialized: now 1790305050401407 us; error 50 us; skew 500 ppm
I20260925 02:57:30.401779 27221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:30 dist-test-slave-8zvk krb5kdc[26823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305050, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:30.404738 27221 init.cc:377] Logged in from keytab as oryx/127.18.196.195@KRBTEST.COM (short username oryx)
I20260925 02:57:30.406189 27221 webserver.cc:460] Webserver started at http://127.18.196.195:39911/ using document root <none> and password file <none>
I20260925 02:57:30.406925 27221 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:30.407078 27221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:30.407400 27221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:30.410151 27221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "7a617a0a6960433e9684dd9896366db6"
format_stamp: "Formatted at 2026-09-25 02:57:30 on dist-test-slave-8zvk"
server_key: "40c4d180bf4a1860d23d3fa22fe74402"
server_key_iv: "55e477edb691d1ee0c3b86c71df04ff8"
server_key_version: "encryptionkey@0"
I20260925 02:57:30.411149 27221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "7a617a0a6960433e9684dd9896366db6"
format_stamp: "Formatted at 2026-09-25 02:57:30 on dist-test-slave-8zvk"
server_key: "40c4d180bf4a1860d23d3fa22fe74402"
server_key_iv: "55e477edb691d1ee0c3b86c71df04ff8"
server_key_version: "encryptionkey@0"
I20260925 02:57:30.416070 27221 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:57:30.419354 27236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:30.420624 27221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:57:30.420800 27221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "7a617a0a6960433e9684dd9896366db6"
format_stamp: "Formatted at 2026-09-25 02:57:30 on dist-test-slave-8zvk"
server_key: "40c4d180bf4a1860d23d3fa22fe74402"
server_key_iv: "55e477edb691d1ee0c3b86c71df04ff8"
server_key_version: "encryptionkey@0"
I20260925 02:57:30.421020 27221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:30.441190 27221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:30.445206 27221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:30.445488 27221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:30.448202 27221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:30.448323 27221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:30.448530 27221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:30.448611 27221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:57:30 dist-test-slave-8zvk krb5kdc[26823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305050, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM
I20260925 02:57:30.474704 26963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:30.448563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:42381 (local address 127.18.196.254:34109)
0925 02:57:30.448778 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:30.448785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:30.450051 (+ 1266us) server_negotiation.cc:407] Connection header received
0925 02:57:30.453061 (+ 3010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:30.453069 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:30.453164 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:30.453271 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:30.456034 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.457015 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.460239 (+ 3224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.460833 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.461002 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:30.466651 (+ 5649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:30.466658 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:30.466663 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:30.466739 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:30.470350 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:30.471413 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:30.471418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:30.471422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:30.471572 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:30.472206 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:30.472209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:30.472211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:30.472620 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:30.472801 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:30.473832 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0925 02:57:30.474155 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 02:57:30.475209 27245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:30.449039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:34109 (local address 127.18.196.195:42381)
0925 02:57:30.449867 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:30.449920 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:30.452712 (+ 2792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:30.453551 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:30.453567 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:30.454533 (+ 966us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:30.455721 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:30.455753 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.457218 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.457225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:30.460036 (+ 2811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:30.460076 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.461157 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.461174 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:30.461354 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:30.462086 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:30.462126 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:30.466250 (+ 4124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:30.470542 (+ 4292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:30.470549 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:30.470561 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:30.471274 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:30.471700 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:30.471707 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:30.471712 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:30.472051 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:30.472779 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:30.472786 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:30.473177 (+ 391us) client_negotiation.cc:769] Sending connection context
0925 02:57:30.473588 (+ 411us) client_negotiation.cc:240] Negotiation successful
0925 02:57:30.474012 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":314,"threads_started":1}
I20260925 02:57:30.497189 27322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:30.481447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:34109 (local address 127.18.196.195:49595)
0925 02:57:30.482095 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:30.482137 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:30.482271 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:30.482675 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:30.482678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:30.483300 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:30.483734 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:30.483756 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.485350 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.485357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:30.486391 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:30.486403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.487563 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.487567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:30.487695 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:30.488314 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:30.488330 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:30.489438 (+ 1108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:30.493946 (+ 4508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:30.493951 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:30.493954 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:30.494503 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:30.495021 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:30.495029 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:30.495032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:30.495189 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:30.496002 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:30.496007 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:30.496209 (+ 202us) client_negotiation.cc:769] Sending connection context
0925 02:57:30.496375 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 02:57:30.496717 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":393,"threads_started":1}
I20260925 02:57:30.497331 26963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:30.481385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:49595 (local address 127.18.196.254:34109)
0925 02:57:30.481570 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:30.481578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:30.482218 (+ 640us) server_negotiation.cc:407] Connection header received
0925 02:57:30.482412 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:30.482420 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:30.482523 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:30.482659 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:30.483911 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.485211 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.486547 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.487399 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.487534 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:30.489685 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:30.489690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:30.489693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:30.489722 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:30.493762 (+ 4040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:30.494644 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:30.494653 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:30.494659 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:30.494885 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:30.495321 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:30.495329 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:30.495335 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:30.495839 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:30.496014 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:30.496533 (+ 519us) server_negotiation.cc:299] Negotiation successful
0925 02:57:30.496845 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:57:30.504878 27077 heartbeater.cc:503] Master 127.18.196.254:34109 was elected leader, sending a full tablet report...
I20260925 02:57:30.505575 27221 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:45013
I20260925 02:57:30.505599 27356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:45013 every 8 connection(s)
I20260925 02:57:30.506940 27221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 02:57:30.508790 27357 heartbeater.cc:347] Connected to a master server at 127.18.196.254:34109
I20260925 02:57:30.509024 27357 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:30.509663 27357 heartbeater.cc:511] Master 127.18.196.254:34109 requested a full tablet report, sending...
I20260925 02:57:30.511004 26867 ts_manager.cc:194] Registered new tserver with Master: 7a617a0a6960433e9684dd9896366db6 (127.18.196.195:45013)
I20260925 02:57:30.511811 26867 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.196.195@KRBTEST.COM'} at 127.18.196.195:42381
I20260925 02:57:30.512168 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 27221
I20260925 02:57:30.512351 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 02:57:30.512832 19219 external_mini_cluster.cc:1428] Setting key 6aeefbaa9560324af817158805cd6e28
I20260925 02:57:30.515555 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:57:30 dist-test-slave-8zvk krb5kdc[26823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305050, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:57:30 dist-test-slave-8zvk krb5kdc[26823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:30 dist-test-slave-8zvk krb5kdc[26823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:57:30.565814 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:30.555948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42308 (local address 127.18.196.254:34109)
0925 02:57:30.556194 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:30.556202 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:30.556357 (+ 155us) server_negotiation.cc:407] Connection header received
0925 02:57:30.556531 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:30.556536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:30.556601 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:30.556683 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:30.557913 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.558612 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.559777 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.560160 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.560249 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:30.565406 (+ 5157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42308: BlockingRecv error: recv got EOF from 127.0.0.1:42308 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 25 02:57:30 dist-test-slave-8zvk krb5kdc[26823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305050, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM
I20260925 02:57:30.582156 26963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:30.569059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42314 (local address 127.18.196.254:34109)
0925 02:57:30.569254 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:30.569262 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:30.569507 (+ 245us) server_negotiation.cc:407] Connection header received
0925 02:57:30.569694 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:30.569698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:30.569765 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:30.569864 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:30.570938 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.571639 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.572871 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.573305 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.573438 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:30.577087 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:30.577092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:30.577095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:30.577126 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:30.579620 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:30.580249 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:30.580255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:30.580259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:30.580386 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:30.580761 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:30.580766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:30.580769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:30.581036 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:30.581339 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:30.581627 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 02:57:30.581848 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:57:30.586203 26867 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:57:30.593055 26867 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:57:30.614982 27376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:30.605124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33577 (local address 127.0.0.1:54192)
0925 02:57:30.606679 (+ 1555us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:30.606700 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:30.606812 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:30.607775 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:30.607782 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:30.607813 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:30.608260 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:30.608280 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.609976 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.609981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:30.612400 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:30.612415 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.613867 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.613873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:30.614037 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:30.614116 (+ 79us) client_negotiation.cc:769] Sending connection context
0925 02:57:30.614256 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 02:57:30.614478 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1432,"mutex_wait_us":2,"thread_start_us":1439,"threads_started":1}
I20260925 02:57:30.615470 27375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:30.605026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:41945 (local address 127.0.0.1:45178)
0925 02:57:30.605888 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:30.605943 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:30.606123 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:30.607833 (+ 1710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:30.607840 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:30.607891 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:30.608421 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:30.608434 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.610350 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.610355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:30.612610 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:30.612619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.614495 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.614503 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:30.614723 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:30.614758 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 02:57:30.614884 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 02:57:30.615034 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":74,"thread_start_us":456,"threads_started":1}
I20260925 02:57:30.615499 27379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:30.606964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:45013 (local address 127.0.0.1:53990)
0925 02:57:30.607488 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:30.607508 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:30.607614 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:30.608852 (+ 1238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:30.608859 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:30.608884 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:30.609212 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:30.609222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.611029 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.611034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:30.612861 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:30.612876 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.614651 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.614660 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:30.614871 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:30.614903 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 02:57:30.614962 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 02:57:30.615062 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":3,"thread_start_us":624,"threads_started":1}
I20260925 02:57:30.615891 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:30.605651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54192 (local address 127.18.196.193:33577)
0925 02:57:30.606731 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:30.606743 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:30.606783 (+ 40us) server_negotiation.cc:407] Connection header received
0925 02:57:30.607035 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:30.607046 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:30.607300 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:30.607494 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:30.608513 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.609839 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.612567 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.613660 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.613849 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:30.614032 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:30.615392 (+ 1360us) server_negotiation.cc:299] Negotiation successful
0925 02:57:30.615586 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":715,"threads_started":1}
I20260925 02:57:30.616210 27377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:30.605258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45178 (local address 127.18.196.194:41945)
0925 02:57:30.607244 (+ 1986us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:30.607255 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:30.607285 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:57:30.607398 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:30.607406 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:30.607620 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:30.608623 (+ 1003us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:30.608846 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.610213 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.612744 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.613996 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.614184 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:30.614340 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:30.615756 (+ 1416us) server_negotiation.cc:299] Negotiation successful
0925 02:57:30.615928 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1835,"thread_start_us":1703,"threads_started":1}
I20260925 02:57:30.616665 27380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:30.607436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.18.196.195:45013)
0925 02:57:30.608220 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:30.608230 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:30.608265 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:57:30.608374 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:30.608380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:30.608609 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:30.608850 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:30.609436 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.610857 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.613022 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.614428 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.615673 (+ 1245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:30.616098 (+ 425us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:30.616180 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 02:57:30.616347 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":1106,"threads_started":1}
I20260925 02:57:30.619776 27287 tablet_service.cc:1467] Processing CreateTablet for tablet cffb4b3db3f841e4aebffd928f894b34 (DEFAULT_TABLE table=test-table [id=7739d01237ae48659c5b3388000aee74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:57:30.619804 27147 tablet_service.cc:1467] Processing CreateTablet for tablet cffb4b3db3f841e4aebffd928f894b34 (DEFAULT_TABLE table=test-table [id=7739d01237ae48659c5b3388000aee74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:57:30.619804 27006 tablet_service.cc:1467] Processing CreateTablet for tablet cffb4b3db3f841e4aebffd928f894b34 (DEFAULT_TABLE table=test-table [id=7739d01237ae48659c5b3388000aee74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:57:30.621084 27287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cffb4b3db3f841e4aebffd928f894b34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:30.621173 27006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cffb4b3db3f841e4aebffd928f894b34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:30.621188 27147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cffb4b3db3f841e4aebffd928f894b34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:30.633975 27381 tablet_bootstrap.cc:492] T cffb4b3db3f841e4aebffd928f894b34 P 7a617a0a6960433e9684dd9896366db6: Bootstrap starting.
I20260925 02:57:30.635864 27383 tablet_bootstrap.cc:492] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce: Bootstrap starting.
I20260925 02:57:30.636566 27382 tablet_bootstrap.cc:492] T cffb4b3db3f841e4aebffd928f894b34 P 45a8bfe724a24cadb6ec77a4a749b7ef: Bootstrap starting.
I20260925 02:57:30.639045 27381 tablet_bootstrap.cc:654] T cffb4b3db3f841e4aebffd928f894b34 P 7a617a0a6960433e9684dd9896366db6: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:30.639447 27383 tablet_bootstrap.cc:654] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:30.639969 27382 tablet_bootstrap.cc:654] T cffb4b3db3f841e4aebffd928f894b34 P 45a8bfe724a24cadb6ec77a4a749b7ef: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:30.640367 27381 log.cc:824] T cffb4b3db3f841e4aebffd928f894b34 P 7a617a0a6960433e9684dd9896366db6: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:30.640575 27383 log.cc:824] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:30.641197 27382 log.cc:824] T cffb4b3db3f841e4aebffd928f894b34 P 45a8bfe724a24cadb6ec77a4a749b7ef: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:30.643675 27381 tablet_bootstrap.cc:492] T cffb4b3db3f841e4aebffd928f894b34 P 7a617a0a6960433e9684dd9896366db6: No bootstrap required, opened a new log
I20260925 02:57:30.643935 27381 ts_tablet_manager.cc:1397] T cffb4b3db3f841e4aebffd928f894b34 P 7a617a0a6960433e9684dd9896366db6: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260925 02:57:30.644110 27382 tablet_bootstrap.cc:492] T cffb4b3db3f841e4aebffd928f894b34 P 45a8bfe724a24cadb6ec77a4a749b7ef: No bootstrap required, opened a new log
I20260925 02:57:30.644388 27382 ts_tablet_manager.cc:1397] T cffb4b3db3f841e4aebffd928f894b34 P 45a8bfe724a24cadb6ec77a4a749b7ef: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 02:57:30.645241 27383 tablet_bootstrap.cc:492] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce: No bootstrap required, opened a new log
I20260925 02:57:30.645547 27383 ts_tablet_manager.cc:1397] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260925 02:57:30.651378 27381 raft_consensus.cc:348] T cffb4b3db3f841e4aebffd928f894b34 P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:30.651795 27381 raft_consensus.cc:374] T cffb4b3db3f841e4aebffd928f894b34 P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:30.651942 27381 raft_consensus.cc:729] T cffb4b3db3f841e4aebffd928f894b34 P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a617a0a6960433e9684dd9896366db6, State: Initialized, Role: FOLLOWER
I20260925 02:57:30.652555 27381 consensus_queue.cc:260] T cffb4b3db3f841e4aebffd928f894b34 P 7a617a0a6960433e9684dd9896366db6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:30.653674 27357 heartbeater.cc:503] Master 127.18.196.254:34109 was elected leader, sending a full tablet report...
I20260925 02:57:30.654228 27381 ts_tablet_manager.cc:1428] T cffb4b3db3f841e4aebffd928f894b34 P 7a617a0a6960433e9684dd9896366db6: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
I20260925 02:57:30.654826 27382 raft_consensus.cc:348] T cffb4b3db3f841e4aebffd928f894b34 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:30.655289 27382 raft_consensus.cc:374] T cffb4b3db3f841e4aebffd928f894b34 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:30.655390 27382 raft_consensus.cc:729] T cffb4b3db3f841e4aebffd928f894b34 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a8bfe724a24cadb6ec77a4a749b7ef, State: Initialized, Role: FOLLOWER
I20260925 02:57:30.655972 27382 consensus_queue.cc:260] T cffb4b3db3f841e4aebffd928f894b34 P 45a8bfe724a24cadb6ec77a4a749b7ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:30.657637 27382 ts_tablet_manager.cc:1428] T cffb4b3db3f841e4aebffd928f894b34 P 45a8bfe724a24cadb6ec77a4a749b7ef: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260925 02:57:30.657478 27383 raft_consensus.cc:348] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:30.657927 27383 raft_consensus.cc:374] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:30.658061 27383 raft_consensus.cc:729] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd65e5165204496787f7e3023d03afce, State: Initialized, Role: FOLLOWER
I20260925 02:57:30.658583 27383 consensus_queue.cc:260] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:30.659888 27217 heartbeater.cc:503] Master 127.18.196.254:34109 was elected leader, sending a full tablet report...
I20260925 02:57:30.660281 27383 ts_tablet_manager.cc:1428] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce: Time spent starting tablet: real 0.015s user 0.011s sys 0.001s
I20260925 02:57:30.662978 27389 raft_consensus.cc:482] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:57:30.663247 27389 raft_consensus.cc:504] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:30.664884 27389 leader_election.cc:288] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45a8bfe724a24cadb6ec77a4a749b7ef (127.18.196.193:33577), 7a617a0a6960433e9684dd9896366db6 (127.18.196.195:45013)
I20260925 02:57:30.671449 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:30.665217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:55369 (local address 127.18.196.193:33577)
0925 02:57:30.665479 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:30.665489 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:30.665998 (+ 509us) server_negotiation.cc:407] Connection header received
0925 02:57:30.666209 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:30.666217 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:30.666329 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:30.666436 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:30.667145 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.667947 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.669687 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.670462 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.670630 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:30.670708 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:30.671079 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 02:57:30.671206 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:57:30.671491 27390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:30.665231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33577 (local address 127.18.196.194:55369)
0925 02:57:30.665870 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:30.665891 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:30.666048 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:30.666592 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:30.666595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:30.666619 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:30.666994 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:30.667008 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.668071 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.668076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:30.669536 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:30.669548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.670676 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.670682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:30.670825 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:30.670932 (+ 107us) client_negotiation.cc:769] Sending connection context
0925 02:57:30.671034 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 02:57:30.671171 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":1,"thread_start_us":314,"threads_started":1}
I20260925 02:57:30.672610 27391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:30.666027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:45013 (local address 127.18.196.194:40787)
0925 02:57:30.666611 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:30.666625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:30.666816 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:30.667153 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:30.667157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:30.667180 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:30.667555 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:30.667563 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.668625 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.668630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:30.670092 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:30.670103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.671887 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.671892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:30.672084 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:30.672115 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 02:57:30.672222 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 02:57:30.672333 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":51,"thread_start_us":310,"threads_started":1}
I20260925 02:57:30.672667 27380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:30.666234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:40787 (local address 127.18.196.195:45013)
0925 02:57:30.666431 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:30.666437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:30.666715 (+ 278us) server_negotiation.cc:407] Connection header received
0925 02:57:30.666970 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:30.666977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:30.667045 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:30.667140 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:30.667709 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.668476 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.670239 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.671157 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.671300 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:30.671363 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:30.672284 (+ 921us) server_negotiation.cc:299] Negotiation successful
0925 02:57:30.672409 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:57:30.672721 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffb4b3db3f841e4aebffd928f894b34" candidate_uuid: "fd65e5165204496787f7e3023d03afce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" is_pre_election: true
I20260925 02:57:30.673234 27031 raft_consensus.cc:2454] T cffb4b3db3f841e4aebffd928f894b34 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd65e5165204496787f7e3023d03afce in term 0.
I20260925 02:57:30.673640 27307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffb4b3db3f841e4aebffd928f894b34" candidate_uuid: "fd65e5165204496787f7e3023d03afce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a617a0a6960433e9684dd9896366db6" is_pre_election: true
I20260925 02:57:30.673902 27100 leader_election.cc:302] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a8bfe724a24cadb6ec77a4a749b7ef, fd65e5165204496787f7e3023d03afce; no voters:
I20260925 02:57:30.674065 27307 raft_consensus.cc:2454] T cffb4b3db3f841e4aebffd928f894b34 P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd65e5165204496787f7e3023d03afce in term 0.
I20260925 02:57:30.674316 27389 raft_consensus.cc:2781] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:57:30.674523 27389 raft_consensus.cc:482] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:57:30.674639 27389 raft_consensus.cc:3037] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:30.676785 27389 raft_consensus.cc:504] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:30.677410 27389 leader_election.cc:288] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce [CANDIDATE]: Term 1 election: Requested vote from peers 45a8bfe724a24cadb6ec77a4a749b7ef (127.18.196.193:33577), 7a617a0a6960433e9684dd9896366db6 (127.18.196.195:45013)
I20260925 02:57:30.677843 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffb4b3db3f841e4aebffd928f894b34" candidate_uuid: "fd65e5165204496787f7e3023d03afce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef"
I20260925 02:57:30.677863 27307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffb4b3db3f841e4aebffd928f894b34" candidate_uuid: "fd65e5165204496787f7e3023d03afce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a617a0a6960433e9684dd9896366db6"
I20260925 02:57:30.678028 27031 raft_consensus.cc:3037] T cffb4b3db3f841e4aebffd928f894b34 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:30.678069 27307 raft_consensus.cc:3037] T cffb4b3db3f841e4aebffd928f894b34 P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:30.680253 27031 raft_consensus.cc:2454] T cffb4b3db3f841e4aebffd928f894b34 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd65e5165204496787f7e3023d03afce in term 1.
I20260925 02:57:30.680398 27307 raft_consensus.cc:2454] T cffb4b3db3f841e4aebffd928f894b34 P 7a617a0a6960433e9684dd9896366db6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd65e5165204496787f7e3023d03afce in term 1.
I20260925 02:57:30.680667 27100 leader_election.cc:302] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a8bfe724a24cadb6ec77a4a749b7ef, fd65e5165204496787f7e3023d03afce; no voters:
I20260925 02:57:30.680931 27389 raft_consensus.cc:2781] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:30.681330 27389 raft_consensus.cc:686] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce [term 1 LEADER]: Becoming Leader. State: Replica: fd65e5165204496787f7e3023d03afce, State: Running, Role: LEADER
I20260925 02:57:30.681871 27389 consensus_queue.cc:237] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:30.686105 26868 catalog_manager.cc:5495] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce reported cstate change: term changed from 0 to 1, leader changed from <none> to fd65e5165204496787f7e3023d03afce (127.18.196.194). New cstate: current_term: 1 leader_uuid: "fd65e5165204496787f7e3023d03afce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } health_report { overall_health: UNKNOWN } } }
I20260925 02:57:30.718400 27377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:30.713426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45190 (local address 127.18.196.194:41945)
0925 02:57:30.713626 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:30.713631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:30.713648 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:57:30.713823 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:30.713827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:30.713909 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:30.714032 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:57:30.714345 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.715113 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.716506 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.716892 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.716983 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:30.717390 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:57:30.717913 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:57:30.718016 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:30.718141 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 02:57:30.718202 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:57:30.726670 27031 raft_consensus.cc:1261] T cffb4b3db3f841e4aebffd928f894b34 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 1 FOLLOWER]: Refusing update from remote peer fd65e5165204496787f7e3023d03afce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:57:30.726778 27307 raft_consensus.cc:1261] T cffb4b3db3f841e4aebffd928f894b34 P 7a617a0a6960433e9684dd9896366db6 [term 1 FOLLOWER]: Refusing update from remote peer fd65e5165204496787f7e3023d03afce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:57:30.727707 27389 consensus_queue.cc:1035] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce [LEADER]: Connected to new peer: Peer: permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:57:30.728041 27392 consensus_queue.cc:1035] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:57:30.738528 27395 mvcc.cc:205] Tried to move back new op lower bound from 7333089487765442560 to 7333089487596658688. Current Snapshot: MvccSnapshot[applied={T|T < 7333089487765442560}]
I20260925 02:57:30.754914 26867 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:57:30.755199 26867 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:57:30.758656 26867 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a: Sending DeleteTablet for 3 replicas of tablet cffb4b3db3f841e4aebffd928f894b34
I20260925 02:57:30.759755 27147 tablet_service.cc:1514] Processing DeleteTablet for tablet cffb4b3db3f841e4aebffd928f894b34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:30 UTC) from {username='oryx', principal='oryx/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:45178
I20260925 02:57:30.759825 27006 tablet_service.cc:1514] Processing DeleteTablet for tablet cffb4b3db3f841e4aebffd928f894b34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:30 UTC) from {username='oryx', principal='oryx/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:54192
I20260925 02:57:30.760474 27287 tablet_service.cc:1514] Processing DeleteTablet for tablet cffb4b3db3f841e4aebffd928f894b34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:30 UTC) from {username='oryx', principal='oryx/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:53990
I20260925 02:57:30.760738 27405 tablet_replica.cc:330] T cffb4b3db3f841e4aebffd928f894b34 P 45a8bfe724a24cadb6ec77a4a749b7ef: stopping tablet replica
I20260925 02:57:30.761137 27405 raft_consensus.cc:2229] T cffb4b3db3f841e4aebffd928f894b34 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:57:30.761613 27405 raft_consensus.cc:2258] T cffb4b3db3f841e4aebffd928f894b34 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:57:30.761981 26867 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 02:57:30.762235 27407 tablet_replica.cc:330] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce: stopping tablet replica
W20260925 02:57:30.762638 26867 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:57:30.762758 27407 raft_consensus.cc:2229] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:57:30.763522 27407 raft_consensus.cc:2258] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:57:30.764456 27408 tablet_replica.cc:330] T cffb4b3db3f841e4aebffd928f894b34 P 7a617a0a6960433e9684dd9896366db6: stopping tablet replica
I20260925 02:57:30.764878 27408 raft_consensus.cc:2229] T cffb4b3db3f841e4aebffd928f894b34 P 7a617a0a6960433e9684dd9896366db6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:57:30.765434 27408 raft_consensus.cc:2258] T cffb4b3db3f841e4aebffd928f894b34 P 7a617a0a6960433e9684dd9896366db6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:57:30.766754 27405 ts_tablet_manager.cc:1905] T cffb4b3db3f841e4aebffd928f894b34 P 45a8bfe724a24cadb6ec77a4a749b7ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:57:30.767063 27407 ts_tablet_manager.cc:1905] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:57:30.767956 27408 ts_tablet_manager.cc:1905] T cffb4b3db3f841e4aebffd928f894b34 P 7a617a0a6960433e9684dd9896366db6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:57:30.771247 27006 tablet_service.cc:1467] Processing CreateTablet for tablet 0454fb6e3de44c7d8c24870ce8ee7f84 (DEFAULT_TABLE table=test-table [id=d35577e7161e41dd9335d98fcb5ca2cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:57:30.771715 27006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0454fb6e3de44c7d8c24870ce8ee7f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:30.773728 27407 ts_tablet_manager.cc:1918] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:57:30.773753 27405 ts_tablet_manager.cc:1918] T cffb4b3db3f841e4aebffd928f894b34 P 45a8bfe724a24cadb6ec77a4a749b7ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:57:30.773882 27405 log.cc:1196] T cffb4b3db3f841e4aebffd928f894b34 P 45a8bfe724a24cadb6ec77a4a749b7ef: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-0/wal/wals/cffb4b3db3f841e4aebffd928f894b34
I20260925 02:57:30.774034 27407 log.cc:1196] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/cffb4b3db3f841e4aebffd928f894b34
I20260925 02:57:30.774405 27405 ts_tablet_manager.cc:1939] T cffb4b3db3f841e4aebffd928f894b34 P 45a8bfe724a24cadb6ec77a4a749b7ef: Deleting consensus metadata
I20260925 02:57:30.774960 27407 ts_tablet_manager.cc:1939] T cffb4b3db3f841e4aebffd928f894b34 P fd65e5165204496787f7e3023d03afce: Deleting consensus metadata
I20260925 02:57:30.775544 27147 tablet_service.cc:1467] Processing CreateTablet for tablet 0454fb6e3de44c7d8c24870ce8ee7f84 (DEFAULT_TABLE table=test-table [id=d35577e7161e41dd9335d98fcb5ca2cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:57:30.775969 27147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0454fb6e3de44c7d8c24870ce8ee7f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:30.776114 26855 catalog_manager.cc:4841] TS 45a8bfe724a24cadb6ec77a4a749b7ef (127.18.196.193:33577): tablet cffb4b3db3f841e4aebffd928f894b34 (table test-table [id=7739d01237ae48659c5b3388000aee74]) successfully deleted
I20260925 02:57:30.777482 27287 tablet_service.cc:1467] Processing CreateTablet for tablet 0454fb6e3de44c7d8c24870ce8ee7f84 (DEFAULT_TABLE table=test-table [id=d35577e7161e41dd9335d98fcb5ca2cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:57:30.777626 26854 catalog_manager.cc:4841] TS fd65e5165204496787f7e3023d03afce (127.18.196.194:41945): tablet cffb4b3db3f841e4aebffd928f894b34 (table test-table [id=7739d01237ae48659c5b3388000aee74]) successfully deleted
I20260925 02:57:30.777997 27287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0454fb6e3de44c7d8c24870ce8ee7f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:30.779695 27382 tablet_bootstrap.cc:492] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 45a8bfe724a24cadb6ec77a4a749b7ef: Bootstrap starting.
I20260925 02:57:30.781131 27382 tablet_bootstrap.cc:654] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 45a8bfe724a24cadb6ec77a4a749b7ef: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:30.784696 27381 tablet_bootstrap.cc:492] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 7a617a0a6960433e9684dd9896366db6: Bootstrap starting.
I20260925 02:57:30.784956 27408 ts_tablet_manager.cc:1918] T cffb4b3db3f841e4aebffd928f894b34 P 7a617a0a6960433e9684dd9896366db6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:57:30.785172 27408 log.cc:1196] T cffb4b3db3f841e4aebffd928f894b34 P 7a617a0a6960433e9684dd9896366db6: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/cffb4b3db3f841e4aebffd928f894b34
I20260925 02:57:30.785774 27408 ts_tablet_manager.cc:1939] T cffb4b3db3f841e4aebffd928f894b34 P 7a617a0a6960433e9684dd9896366db6: Deleting consensus metadata
I20260925 02:57:30.786083 27382 tablet_bootstrap.cc:492] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 45a8bfe724a24cadb6ec77a4a749b7ef: No bootstrap required, opened a new log
I20260925 02:57:30.786259 27382 ts_tablet_manager.cc:1397] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 45a8bfe724a24cadb6ec77a4a749b7ef: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260925 02:57:30.786373 27381 tablet_bootstrap.cc:654] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 7a617a0a6960433e9684dd9896366db6: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:30.787153 27382 raft_consensus.cc:348] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:30.787441 27382 raft_consensus.cc:374] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:30.787590 27382 raft_consensus.cc:729] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a8bfe724a24cadb6ec77a4a749b7ef, State: Initialized, Role: FOLLOWER
I20260925 02:57:30.787719 26855 catalog_manager.cc:4841] TS 7a617a0a6960433e9684dd9896366db6 (127.18.196.195:45013): tablet cffb4b3db3f841e4aebffd928f894b34 (table test-table [id=7739d01237ae48659c5b3388000aee74]) successfully deleted
I20260925 02:57:30.788021 27382 consensus_queue.cc:260] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 45a8bfe724a24cadb6ec77a4a749b7ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:30.788831 27382 ts_tablet_manager.cc:1428] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 45a8bfe724a24cadb6ec77a4a749b7ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:57:30.790513 27383 tablet_bootstrap.cc:492] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce: Bootstrap starting.
I20260925 02:57:30.791998 27381 tablet_bootstrap.cc:492] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 7a617a0a6960433e9684dd9896366db6: No bootstrap required, opened a new log
I20260925 02:57:30.792197 27381 ts_tablet_manager.cc:1397] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 7a617a0a6960433e9684dd9896366db6: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260925 02:57:30.792520 27383 tablet_bootstrap.cc:654] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:30.793138 27381 raft_consensus.cc:348] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:30.793421 27381 raft_consensus.cc:374] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:30.793574 27381 raft_consensus.cc:729] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a617a0a6960433e9684dd9896366db6, State: Initialized, Role: FOLLOWER
I20260925 02:57:30.793918 27381 consensus_queue.cc:260] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 7a617a0a6960433e9684dd9896366db6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:30.794744 27381 ts_tablet_manager.cc:1428] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 7a617a0a6960433e9684dd9896366db6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 02:57:30.796069 27383 tablet_bootstrap.cc:492] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce: No bootstrap required, opened a new log
I20260925 02:57:30.796249 27383 ts_tablet_manager.cc:1397] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:57:30.797163 27383 raft_consensus.cc:348] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:30.797365 27383 raft_consensus.cc:374] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:30.797478 27383 raft_consensus.cc:729] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd65e5165204496787f7e3023d03afce, State: Initialized, Role: FOLLOWER
I20260925 02:57:30.797735 27383 consensus_queue.cc:260] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:30.798903 27383 ts_tablet_manager.cc:1428] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 02:57:30.808039 27392 raft_consensus.cc:482] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:57:30.808279 27392 raft_consensus.cc:504] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:30.808983 27392 leader_election.cc:288] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45a8bfe724a24cadb6ec77a4a749b7ef (127.18.196.193:33577), 7a617a0a6960433e9684dd9896366db6 (127.18.196.195:45013)
I20260925 02:57:30.809566 27307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0454fb6e3de44c7d8c24870ce8ee7f84" candidate_uuid: "fd65e5165204496787f7e3023d03afce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a617a0a6960433e9684dd9896366db6" is_pre_election: true
I20260925 02:57:30.809603 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0454fb6e3de44c7d8c24870ce8ee7f84" candidate_uuid: "fd65e5165204496787f7e3023d03afce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" is_pre_election: true
I20260925 02:57:30.809808 27307 raft_consensus.cc:2454] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd65e5165204496787f7e3023d03afce in term 0.
I20260925 02:57:30.809865 27031 raft_consensus.cc:2454] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd65e5165204496787f7e3023d03afce in term 0.
I20260925 02:57:30.810302 27100 leader_election.cc:302] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a8bfe724a24cadb6ec77a4a749b7ef, fd65e5165204496787f7e3023d03afce; no voters:
I20260925 02:57:30.810647 27392 raft_consensus.cc:2781] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:57:30.810767 27392 raft_consensus.cc:482] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:57:30.810926 27392 raft_consensus.cc:3037] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:30.813616 27392 raft_consensus.cc:504] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:30.814297 27392 leader_election.cc:288] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce [CANDIDATE]: Term 1 election: Requested vote from peers 45a8bfe724a24cadb6ec77a4a749b7ef (127.18.196.193:33577), 7a617a0a6960433e9684dd9896366db6 (127.18.196.195:45013)
I20260925 02:57:30.814939 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0454fb6e3de44c7d8c24870ce8ee7f84" candidate_uuid: "fd65e5165204496787f7e3023d03afce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef"
I20260925 02:57:30.815145 27031 raft_consensus.cc:3037] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:30.818118 27031 raft_consensus.cc:2454] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd65e5165204496787f7e3023d03afce in term 1.
I20260925 02:57:30.818738 27100 leader_election.cc:302] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a8bfe724a24cadb6ec77a4a749b7ef, fd65e5165204496787f7e3023d03afce; no voters:
I20260925 02:57:30.819062 27307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0454fb6e3de44c7d8c24870ce8ee7f84" candidate_uuid: "fd65e5165204496787f7e3023d03afce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a617a0a6960433e9684dd9896366db6"
I20260925 02:57:30.819263 27307 raft_consensus.cc:3037] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:30.819245 27392 raft_consensus.cc:2781] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:30.819502 27392 raft_consensus.cc:686] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce [term 1 LEADER]: Becoming Leader. State: Replica: fd65e5165204496787f7e3023d03afce, State: Running, Role: LEADER
I20260925 02:57:30.819842 27392 consensus_queue.cc:237] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:30.821617 27307 raft_consensus.cc:2454] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 7a617a0a6960433e9684dd9896366db6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd65e5165204496787f7e3023d03afce in term 1.
I20260925 02:57:30.822892 26867 catalog_manager.cc:5495] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce reported cstate change: term changed from 0 to 1, leader changed from <none> to fd65e5165204496787f7e3023d03afce (127.18.196.194). New cstate: current_term: 1 leader_uuid: "fd65e5165204496787f7e3023d03afce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } health_report { overall_health: UNKNOWN } } }
Sep 25 02:57:30 dist-test-slave-8zvk krb5kdc[26823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305048, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.254@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM
I20260925 02:57:30.868443 27418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:30.853237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:34109 (local address 127.0.0.1:42326)
0925 02:57:30.853800 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:30.853813 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:30.853934 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:30.854467 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:30.854474 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:30.855225 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:30.855590 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:30.855601 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.857037 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.857041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:30.857974 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:30.857997 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.858983 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.858988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:30.859105 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:30.859784 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:30.859802 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:30.862095 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:30.865850 (+ 3755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:30.865860 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:30.865865 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:30.866294 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:30.866731 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:30.866734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:30.866736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:30.866947 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:30.867660 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:30.867668 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:30.867816 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 02:57:30.867946 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 02:57:30.868118 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":340,"threads_started":1}
I20260925 02:57:30.868728 26963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:30.853168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42326 (local address 127.18.196.254:34109)
0925 02:57:30.853379 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:30.853383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:30.853894 (+ 511us) server_negotiation.cc:407] Connection header received
0925 02:57:30.854088 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:30.854094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:30.854173 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:30.854280 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:30.855728 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.856919 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.858113 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:30.858718 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:30.858857 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:30.862281 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:30.862288 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:30.862293 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:30.862331 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:30.865735 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:30.866461 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:30.866467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:30.866471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:30.866614 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:30.867049 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:30.867054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:30.867058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:30.867437 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:30.867646 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:30.868074 (+ 428us) server_negotiation.cc:299] Negotiation successful
0925 02:57:30.868294 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260925 02:57:30.873575 26867 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:42326:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 02:57:30.874624 26867 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:57:30.881317 27006 tablet_service.cc:1467] Processing CreateTablet for tablet 312b3e94744e4987bfbf9f12d1260fcf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b1b7ac11126402da30378a70d31b741]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:57:30.881459 27147 tablet_service.cc:1467] Processing CreateTablet for tablet 312b3e94744e4987bfbf9f12d1260fcf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b1b7ac11126402da30378a70d31b741]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:57:30.881634 27287 tablet_service.cc:1467] Processing CreateTablet for tablet 312b3e94744e4987bfbf9f12d1260fcf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b1b7ac11126402da30378a70d31b741]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:57:30.881907 27006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 312b3e94744e4987bfbf9f12d1260fcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:30.882041 27147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 312b3e94744e4987bfbf9f12d1260fcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:30.882140 27287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 312b3e94744e4987bfbf9f12d1260fcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:30.889508 27383 tablet_bootstrap.cc:492] T 312b3e94744e4987bfbf9f12d1260fcf P fd65e5165204496787f7e3023d03afce: Bootstrap starting.
I20260925 02:57:30.890008 27381 tablet_bootstrap.cc:492] T 312b3e94744e4987bfbf9f12d1260fcf P 7a617a0a6960433e9684dd9896366db6: Bootstrap starting.
I20260925 02:57:30.891484 27383 tablet_bootstrap.cc:654] T 312b3e94744e4987bfbf9f12d1260fcf P fd65e5165204496787f7e3023d03afce: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:30.891769 27381 tablet_bootstrap.cc:654] T 312b3e94744e4987bfbf9f12d1260fcf P 7a617a0a6960433e9684dd9896366db6: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:30.898564 27382 tablet_bootstrap.cc:492] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef: Bootstrap starting.
I20260925 02:57:30.900498 27382 tablet_bootstrap.cc:654] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:30.901892 27381 tablet_bootstrap.cc:492] T 312b3e94744e4987bfbf9f12d1260fcf P 7a617a0a6960433e9684dd9896366db6: No bootstrap required, opened a new log
I20260925 02:57:30.902081 27381 ts_tablet_manager.cc:1397] T 312b3e94744e4987bfbf9f12d1260fcf P 7a617a0a6960433e9684dd9896366db6: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260925 02:57:30.902768 27381 raft_consensus.cc:348] T 312b3e94744e4987bfbf9f12d1260fcf P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:30.903059 27381 raft_consensus.cc:374] T 312b3e94744e4987bfbf9f12d1260fcf P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:30.903148 27381 raft_consensus.cc:729] T 312b3e94744e4987bfbf9f12d1260fcf P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a617a0a6960433e9684dd9896366db6, State: Initialized, Role: FOLLOWER
I20260925 02:57:30.903475 27381 consensus_queue.cc:260] T 312b3e94744e4987bfbf9f12d1260fcf P 7a617a0a6960433e9684dd9896366db6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:30.903900 27387 tablet_replica.cc:439] T 312b3e94744e4987bfbf9f12d1260fcf P 7a617a0a6960433e9684dd9896366db6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } } }
I20260925 02:57:30.904191 27387 tablet_replica.cc:442] T 312b3e94744e4987bfbf9f12d1260fcf P 7a617a0a6960433e9684dd9896366db6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:57:30.906904 27381 ts_tablet_manager.cc:1428] T 312b3e94744e4987bfbf9f12d1260fcf P 7a617a0a6960433e9684dd9896366db6: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260925 02:57:30.909631 27383 tablet_bootstrap.cc:492] T 312b3e94744e4987bfbf9f12d1260fcf P fd65e5165204496787f7e3023d03afce: No bootstrap required, opened a new log
I20260925 02:57:30.909807 27383 ts_tablet_manager.cc:1397] T 312b3e94744e4987bfbf9f12d1260fcf P fd65e5165204496787f7e3023d03afce: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.010s
I20260925 02:57:30.910555 27383 raft_consensus.cc:348] T 312b3e94744e4987bfbf9f12d1260fcf P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:30.910778 27383 raft_consensus.cc:374] T 312b3e94744e4987bfbf9f12d1260fcf P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:30.910866 27383 raft_consensus.cc:729] T 312b3e94744e4987bfbf9f12d1260fcf P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd65e5165204496787f7e3023d03afce, State: Initialized, Role: FOLLOWER
I20260925 02:57:30.911055 27382 tablet_bootstrap.cc:492] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef: No bootstrap required, opened a new log
I20260925 02:57:30.911197 27382 ts_tablet_manager.cc:1397] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260925 02:57:30.911120 27383 consensus_queue.cc:260] T 312b3e94744e4987bfbf9f12d1260fcf P fd65e5165204496787f7e3023d03afce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:30.911677 27383 ts_tablet_manager.cc:1428] T 312b3e94744e4987bfbf9f12d1260fcf P fd65e5165204496787f7e3023d03afce: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:57:30.911559 27392 tablet_replica.cc:439] T 312b3e94744e4987bfbf9f12d1260fcf P fd65e5165204496787f7e3023d03afce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } } }
I20260925 02:57:30.911821 27392 tablet_replica.cc:442] T 312b3e94744e4987bfbf9f12d1260fcf P fd65e5165204496787f7e3023d03afce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:57:30.913033 27382 raft_consensus.cc:348] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:30.913280 27382 raft_consensus.cc:374] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:30.913354 27382 raft_consensus.cc:729] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a8bfe724a24cadb6ec77a4a749b7ef, State: Initialized, Role: FOLLOWER
I20260925 02:57:30.913645 27382 consensus_queue.cc:260] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:30.914251 27388 tablet_replica.cc:439] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } } }
I20260925 02:57:30.914614 27388 tablet_replica.cc:442] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:57:30.915439 27382 ts_tablet_manager.cc:1428] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20260925 02:57:30.999542 27078 tablet.cc:2369] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 45a8bfe724a24cadb6ec77a4a749b7ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:57:31.004170 27218 tablet.cc:2369] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:57:31.009469 27358 tablet.cc:2369] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 7a617a0a6960433e9684dd9896366db6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:57:31.045492 27388 raft_consensus.cc:482] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:57:31.045754 27388 raft_consensus.cc:504] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:31.047678 27388 leader_election.cc:288] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd65e5165204496787f7e3023d03afce (127.18.196.194:41945), 7a617a0a6960433e9684dd9896366db6 (127.18.196.195:45013)
I20260925 02:57:31.056414 27377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:31.048222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:40527 (local address 127.18.196.194:41945)
0925 02:57:31.048416 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:31.048422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:31.049285 (+ 863us) server_negotiation.cc:407] Connection header received
0925 02:57:31.049551 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:31.049557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:31.049631 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:31.049721 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:31.050562 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.051371 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.054277 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.055369 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.055547 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:31.055629 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:31.055940 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 02:57:31.056097 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 02:57:31.056485 27423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:31.047972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:41945 (local address 127.18.196.193:40527)
0925 02:57:31.049167 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:31.049187 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:31.049342 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:31.049828 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:31.049832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:31.049857 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:31.050388 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:31.050406 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.051500 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.051509 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:31.054077 (+ 2568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:31.054089 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.055574 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.055580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:31.055769 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:31.055793 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 02:57:31.055915 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 02:57:31.056076 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1066,"mutex_wait_us":299,"thread_start_us":607,"threads_started":1}
I20260925 02:57:31.056761 27380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:31.048245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:55385 (local address 127.18.196.195:45013)
0925 02:57:31.048458 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:31.048467 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:31.049158 (+ 691us) server_negotiation.cc:407] Connection header received
0925 02:57:31.049437 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:31.049444 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:31.049537 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:31.049635 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:31.050378 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.051525 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.054197 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.055485 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.055647 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:31.055735 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:31.056332 (+ 597us) server_negotiation.cc:299] Negotiation successful
0925 02:57:31.056487 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 02:57:31.057493 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:31.047972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:45013 (local address 127.18.196.193:55385)
0925 02:57:31.049024 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:31.049045 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:31.049209 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:31.049663 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:31.049668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:31.049693 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:31.050148 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:31.050166 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.051672 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.051680 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:31.054013 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:31.054030 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.055902 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.055909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:31.056127 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:31.056166 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 02:57:31.056945 (+ 779us) client_negotiation.cc:240] Negotiation successful
0925 02:57:31.057089 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"mutex_wait_us":92,"thread_start_us":648,"threads_started":1}
I20260925 02:57:31.057538 27167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "312b3e94744e4987bfbf9f12d1260fcf" candidate_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd65e5165204496787f7e3023d03afce" is_pre_election: true
I20260925 02:57:31.058058 27167 raft_consensus.cc:2454] T 312b3e94744e4987bfbf9f12d1260fcf P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45a8bfe724a24cadb6ec77a4a749b7ef in term 0.
I20260925 02:57:31.058123 27307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "312b3e94744e4987bfbf9f12d1260fcf" candidate_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a617a0a6960433e9684dd9896366db6" is_pre_election: true
I20260925 02:57:31.058363 27307 raft_consensus.cc:2454] T 312b3e94744e4987bfbf9f12d1260fcf P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45a8bfe724a24cadb6ec77a4a749b7ef in term 0.
I20260925 02:57:31.058754 26957 leader_election.cc:302] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a8bfe724a24cadb6ec77a4a749b7ef, fd65e5165204496787f7e3023d03afce; no voters:
I20260925 02:57:31.059219 27388 raft_consensus.cc:2781] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:57:31.059347 27388 raft_consensus.cc:482] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:57:31.059444 27388 raft_consensus.cc:3037] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:31.061890 27388 raft_consensus.cc:504] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:31.062595 27388 leader_election.cc:288] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef [CANDIDATE]: Term 1 election: Requested vote from peers fd65e5165204496787f7e3023d03afce (127.18.196.194:41945), 7a617a0a6960433e9684dd9896366db6 (127.18.196.195:45013)
I20260925 02:57:31.063060 27167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "312b3e94744e4987bfbf9f12d1260fcf" candidate_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd65e5165204496787f7e3023d03afce"
I20260925 02:57:31.063140 27307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "312b3e94744e4987bfbf9f12d1260fcf" candidate_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a617a0a6960433e9684dd9896366db6"
I20260925 02:57:31.063284 27167 raft_consensus.cc:3037] T 312b3e94744e4987bfbf9f12d1260fcf P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:31.063349 27307 raft_consensus.cc:3037] T 312b3e94744e4987bfbf9f12d1260fcf P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:31.066345 27307 raft_consensus.cc:2454] T 312b3e94744e4987bfbf9f12d1260fcf P 7a617a0a6960433e9684dd9896366db6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45a8bfe724a24cadb6ec77a4a749b7ef in term 1.
I20260925 02:57:31.066593 27167 raft_consensus.cc:2454] T 312b3e94744e4987bfbf9f12d1260fcf P fd65e5165204496787f7e3023d03afce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45a8bfe724a24cadb6ec77a4a749b7ef in term 1.
I20260925 02:57:31.066871 26958 leader_election.cc:302] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a8bfe724a24cadb6ec77a4a749b7ef, 7a617a0a6960433e9684dd9896366db6; no voters:
I20260925 02:57:31.067198 27388 raft_consensus.cc:2781] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:31.067685 27388 raft_consensus.cc:686] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef [term 1 LEADER]: Becoming Leader. State: Replica: 45a8bfe724a24cadb6ec77a4a749b7ef, State: Running, Role: LEADER
I20260925 02:57:31.068172 27388 consensus_queue.cc:237] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:31.070703 27424 tablet_replica.cc:439] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef: TxnStatusTablet state changed. Reason: New leader 45a8bfe724a24cadb6ec77a4a749b7ef. Latest consensus state: current_term: 1 leader_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } } }
I20260925 02:57:31.071123 27424 tablet_replica.cc:442] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:57:31.071820 27426 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 02:57:31.072029 27426 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 02:57:31.072188 26866 catalog_manager.cc:5495] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 45a8bfe724a24cadb6ec77a4a749b7ef (127.18.196.193). New cstate: current_term: 1 leader_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } health_report { overall_health: UNKNOWN } } }
I20260925 02:57:31.116679 27418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:31.108579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33577 (local address 127.0.0.1:54198)
0925 02:57:31.108999 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:31.109022 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:31.109162 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:31.109958 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:31.109964 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:31.109978 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:57:31.110185 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:31.110197 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.111852 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.111858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:31.113496 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:31.113513 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.114505 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.114513 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:31.114733 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:31.114820 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:57:31.116019 (+ 1199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:57:31.116049 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 02:57:31.116143 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 02:57:31.116331 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239}
I20260925 02:57:31.117333 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:31.108582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54198 (local address 127.18.196.193:33577)
0925 02:57:31.109470 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:31.109479 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:31.109505 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:57:31.109624 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:31.109630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:31.109742 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:31.109889 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:57:31.110356 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.111680 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.113668 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.114327 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.114461 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:31.115076 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:57:31.115816 (+ 740us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:57:31.115971 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:31.116927 (+ 956us) server_negotiation.cc:299] Negotiation successful
0925 02:57:31.117019 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746}
I20260925 02:57:31.121887 27307 raft_consensus.cc:1261] T 312b3e94744e4987bfbf9f12d1260fcf P 7a617a0a6960433e9684dd9896366db6 [term 1 FOLLOWER]: Refusing update from remote peer 45a8bfe724a24cadb6ec77a4a749b7ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:57:31.122314 27167 raft_consensus.cc:1261] T 312b3e94744e4987bfbf9f12d1260fcf P fd65e5165204496787f7e3023d03afce [term 1 FOLLOWER]: Refusing update from remote peer 45a8bfe724a24cadb6ec77a4a749b7ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:57:31.122780 27388 consensus_queue.cc:1035] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 02:57:31.123430 27424 consensus_queue.cc:1035] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:57:31.132604 27387 tablet_replica.cc:439] T 312b3e94744e4987bfbf9f12d1260fcf P 7a617a0a6960433e9684dd9896366db6: TxnStatusTablet state changed. Reason: New leader 45a8bfe724a24cadb6ec77a4a749b7ef. Latest consensus state: current_term: 1 leader_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } } }
I20260925 02:57:31.132884 27387 tablet_replica.cc:442] T 312b3e94744e4987bfbf9f12d1260fcf P 7a617a0a6960433e9684dd9896366db6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:57:31.140901 27392 tablet_replica.cc:439] T 312b3e94744e4987bfbf9f12d1260fcf P fd65e5165204496787f7e3023d03afce: TxnStatusTablet state changed. Reason: New leader 45a8bfe724a24cadb6ec77a4a749b7ef. Latest consensus state: current_term: 1 leader_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } } }
I20260925 02:57:31.141216 27392 tablet_replica.cc:442] T 312b3e94744e4987bfbf9f12d1260fcf P fd65e5165204496787f7e3023d03afce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:57:31.148243 27392 tablet_replica.cc:439] T 312b3e94744e4987bfbf9f12d1260fcf P fd65e5165204496787f7e3023d03afce: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } } }
I20260925 02:57:31.148554 27392 tablet_replica.cc:442] T 312b3e94744e4987bfbf9f12d1260fcf P fd65e5165204496787f7e3023d03afce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:57:31.159900 27433 tablet_replica.cc:439] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } } }
I20260925 02:57:31.160264 27433 tablet_replica.cc:442] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:57:31.168212 27387 tablet_replica.cc:439] T 312b3e94744e4987bfbf9f12d1260fcf P 7a617a0a6960433e9684dd9896366db6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } } }
I20260925 02:57:31.168493 27387 tablet_replica.cc:442] T 312b3e94744e4987bfbf9f12d1260fcf P 7a617a0a6960433e9684dd9896366db6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:57:31.179733 27424 tablet_replica.cc:439] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } } }
I20260925 02:57:31.180562 27424 tablet_replica.cc:442] T 312b3e94744e4987bfbf9f12d1260fcf P 45a8bfe724a24cadb6ec77a4a749b7ef: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:57:31.193341 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:31.183529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:43833 (local address 127.18.196.193:33577)
0925 02:57:31.183719 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:31.183726 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:31.186508 (+ 2782us) server_negotiation.cc:407] Connection header received
0925 02:57:31.186822 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:31.186828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:31.186957 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:31.187026 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:57:31.187782 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.188970 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.191170 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.191854 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.191981 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:31.192534 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:57:31.192677 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:57:31.192766 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:31.192946 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 02:57:31.193027 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:57:31.193718 27436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:31.179915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33577 (local address 127.18.196.194:43833)
0925 02:57:31.186398 (+ 6483us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:31.186424 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:31.186641 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:31.187242 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:31.187247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:31.187259 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:57:31.187571 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:31.187585 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.189115 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.189121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:31.191018 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:31.191038 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.192061 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.192070 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:31.192266 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:31.192342 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:57:31.192787 (+ 445us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:57:31.192800 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 02:57:31.192904 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 02:57:31.193073 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6297,"thread_start_us":7137,"threads_started":1}
I20260925 02:57:31.205992 27307 raft_consensus.cc:1261] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 7a617a0a6960433e9684dd9896366db6 [term 1 FOLLOWER]: Refusing update from remote peer fd65e5165204496787f7e3023d03afce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:57:31.206195 27031 raft_consensus.cc:1261] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 1 FOLLOWER]: Refusing update from remote peer fd65e5165204496787f7e3023d03afce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:57:31.206774 27389 consensus_queue.cc:1035] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce [LEADER]: Connected to new peer: Peer: permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:57:31.207095 27392 consensus_queue.cc:1035] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:57:31.236810 27397 mvcc.cc:205] Tried to move back new op lower bound from 7333089489733738496 to 7333089488161472512. Current Snapshot: MvccSnapshot[applied={T|T < 7333089489733738496}]
I20260925 02:57:31.263288 27377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:31.252718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:44411 (local address 127.18.196.194:41945)
0925 02:57:31.252917 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:31.252924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:31.256165 (+ 3241us) server_negotiation.cc:407] Connection header received
0925 02:57:31.256360 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:31.256366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:31.256469 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:31.256571 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:57:31.257480 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.258763 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.261027 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.261812 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.261942 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:31.262496 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:57:31.262628 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:57:31.262684 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:31.262923 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 02:57:31.263005 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:57:31.264137 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:31.253071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:41945 (local address 127.18.196.193:44411)
0925 02:57:31.256034 (+ 2963us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:31.256058 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:31.256207 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:31.256785 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:31.256790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:31.256805 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:57:31.257239 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:31.257252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.258984 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.258990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:31.260870 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:31.260884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.262009 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.262015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:31.262209 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:31.262287 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:57:31.262765 (+ 478us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:57:31.262774 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 02:57:31.263576 (+ 802us) client_negotiation.cc:240] Negotiation successful
0925 02:57:31.263729 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2756,"thread_start_us":2603,"threads_started":1}
I20260925 02:57:31.305207 26866 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:57:31.305420 26866 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:57:31.308403 26866 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a: Sending DeleteTablet for 3 replicas of tablet 0454fb6e3de44c7d8c24870ce8ee7f84
I20260925 02:57:31.309163 27006 tablet_service.cc:1514] Processing DeleteTablet for tablet 0454fb6e3de44c7d8c24870ce8ee7f84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:31 UTC) from {username='oryx', principal='oryx/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:54192
I20260925 02:57:31.309413 27147 tablet_service.cc:1514] Processing DeleteTablet for tablet 0454fb6e3de44c7d8c24870ce8ee7f84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:31 UTC) from {username='oryx', principal='oryx/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:45178
I20260925 02:57:31.309464 27287 tablet_service.cc:1514] Processing DeleteTablet for tablet 0454fb6e3de44c7d8c24870ce8ee7f84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:31 UTC) from {username='oryx', principal='oryx/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:53990
I20260925 02:57:31.310089 27444 tablet_replica.cc:330] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 7a617a0a6960433e9684dd9896366db6: stopping tablet replica
I20260925 02:57:31.310397 27444 raft_consensus.cc:2229] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 7a617a0a6960433e9684dd9896366db6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:57:31.310798 27444 raft_consensus.cc:2258] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 7a617a0a6960433e9684dd9896366db6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:57:31.312186 27444 ts_tablet_manager.cc:1905] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 7a617a0a6960433e9684dd9896366db6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:57:31.321169 27442 tablet_replica.cc:330] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 45a8bfe724a24cadb6ec77a4a749b7ef: stopping tablet replica
I20260925 02:57:31.321424 27442 raft_consensus.cc:2229] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:57:31.321635 27444 ts_tablet_manager.cc:1918] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 7a617a0a6960433e9684dd9896366db6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:57:31.321730 27442 raft_consensus.cc:2258] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:57:31.321857 27444 log.cc:1196] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 7a617a0a6960433e9684dd9896366db6: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/0454fb6e3de44c7d8c24870ce8ee7f84
I20260925 02:57:31.322419 27444 ts_tablet_manager.cc:1939] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 7a617a0a6960433e9684dd9896366db6: Deleting consensus metadata
I20260925 02:57:31.323011 27442 ts_tablet_manager.cc:1905] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 45a8bfe724a24cadb6ec77a4a749b7ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:57:31.323724 26855 catalog_manager.cc:4841] TS 7a617a0a6960433e9684dd9896366db6 (127.18.196.195:45013): tablet 0454fb6e3de44c7d8c24870ce8ee7f84 (table test-table [id=d35577e7161e41dd9335d98fcb5ca2cf]) successfully deleted
I20260925 02:57:31.328836 27442 ts_tablet_manager.cc:1918] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 45a8bfe724a24cadb6ec77a4a749b7ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:57:31.328979 27442 log.cc:1196] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 45a8bfe724a24cadb6ec77a4a749b7ef: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-0/wal/wals/0454fb6e3de44c7d8c24870ce8ee7f84
I20260925 02:57:31.329416 27442 ts_tablet_manager.cc:1939] T 0454fb6e3de44c7d8c24870ce8ee7f84 P 45a8bfe724a24cadb6ec77a4a749b7ef: Deleting consensus metadata
I20260925 02:57:31.330332 26855 catalog_manager.cc:4841] TS 45a8bfe724a24cadb6ec77a4a749b7ef (127.18.196.193:33577): tablet 0454fb6e3de44c7d8c24870ce8ee7f84 (table test-table [id=d35577e7161e41dd9335d98fcb5ca2cf]) successfully deleted
I20260925 02:57:31.331275 27443 tablet_replica.cc:330] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce: stopping tablet replica
I20260925 02:57:31.331530 27443 raft_consensus.cc:2229] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:57:31.331988 27443 raft_consensus.cc:2258] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:57:31.332883 27443 ts_tablet_manager.cc:1905] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:57:31.337497 27443 ts_tablet_manager.cc:1918] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:57:31.337654 27443 log.cc:1196] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/0454fb6e3de44c7d8c24870ce8ee7f84
I20260925 02:57:31.338045 27443 ts_tablet_manager.cc:1939] T 0454fb6e3de44c7d8c24870ce8ee7f84 P fd65e5165204496787f7e3023d03afce: Deleting consensus metadata
I20260925 02:57:31.338948 26854 catalog_manager.cc:4841] TS fd65e5165204496787f7e3023d03afce (127.18.196.194:41945): tablet 0454fb6e3de44c7d8c24870ce8ee7f84 (table test-table [id=d35577e7161e41dd9335d98fcb5ca2cf]) successfully deleted
Sep 25 02:57:31 dist-test-slave-8zvk krb5kdc[26823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:57:31 dist-test-slave-8zvk krb5kdc[26823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:31 dist-test-slave-8zvk krb5kdc[26823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:57:31.381851 27452 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:31.370089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42328 (local address 127.18.196.254:34109)
0925 02:57:31.370963 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:31.370968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:31.370987 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:57:31.371042 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:31.371046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:31.371107 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:31.371197 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:31.372908 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.373633 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.375544 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.375960 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.376086 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:31.381520 (+ 5434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42328: BlockingRecv error: recv got EOF from 127.0.0.1:42328 (error 108)
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":595,"threads_started":1}
Sep 25 02:57:31 dist-test-slave-8zvk krb5kdc[26823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM
I20260925 02:57:31.400458 27452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:31.385399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42338 (local address 127.18.196.254:34109)
0925 02:57:31.385601 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:31.385606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:31.385855 (+ 249us) server_negotiation.cc:407] Connection header received
0925 02:57:31.386018 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:31.386022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:31.386082 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:31.386161 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:31.387701 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.388405 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.390131 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.390507 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.390594 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:31.394907 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:31.394912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:31.394915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:31.394946 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:31.397875 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:31.398505 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:31.398511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:31.398514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:31.398651 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:31.399087 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:31.399093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:31.399097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:31.399348 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:31.399610 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:31.399918 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 02:57:31.400157 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:57:31.404346 26866 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:57:31.405026 26866 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:57:31.411800 27006 tablet_service.cc:1467] Processing CreateTablet for tablet 0ac1b9484d30407c94fa0886d317d274 (DEFAULT_TABLE table=test-table [id=51cb89fef9124970aa81c4b29bccbcbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:57:31.412225 27006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ac1b9484d30407c94fa0886d317d274. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:31.412186 27147 tablet_service.cc:1467] Processing CreateTablet for tablet 0ac1b9484d30407c94fa0886d317d274 (DEFAULT_TABLE table=test-table [id=51cb89fef9124970aa81c4b29bccbcbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:57:31.412631 27147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ac1b9484d30407c94fa0886d317d274. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:31.419255 27458 tablet_bootstrap.cc:492] T 0ac1b9484d30407c94fa0886d317d274 P fd65e5165204496787f7e3023d03afce: Bootstrap starting.
I20260925 02:57:31.419632 27287 tablet_service.cc:1467] Processing CreateTablet for tablet 0ac1b9484d30407c94fa0886d317d274 (DEFAULT_TABLE table=test-table [id=51cb89fef9124970aa81c4b29bccbcbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:57:31.420086 27287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ac1b9484d30407c94fa0886d317d274. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:31.421561 27458 tablet_bootstrap.cc:654] T 0ac1b9484d30407c94fa0886d317d274 P fd65e5165204496787f7e3023d03afce: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:31.423760 27458 tablet_bootstrap.cc:492] T 0ac1b9484d30407c94fa0886d317d274 P fd65e5165204496787f7e3023d03afce: No bootstrap required, opened a new log
I20260925 02:57:31.423910 27458 ts_tablet_manager.cc:1397] T 0ac1b9484d30407c94fa0886d317d274 P fd65e5165204496787f7e3023d03afce: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 02:57:31.425139 27458 raft_consensus.cc:348] T 0ac1b9484d30407c94fa0886d317d274 P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:31.425653 27458 raft_consensus.cc:374] T 0ac1b9484d30407c94fa0886d317d274 P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:31.425755 27458 raft_consensus.cc:729] T 0ac1b9484d30407c94fa0886d317d274 P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd65e5165204496787f7e3023d03afce, State: Initialized, Role: FOLLOWER
I20260925 02:57:31.426280 27458 consensus_queue.cc:260] T 0ac1b9484d30407c94fa0886d317d274 P fd65e5165204496787f7e3023d03afce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:31.427914 27458 ts_tablet_manager.cc:1428] T 0ac1b9484d30407c94fa0886d317d274 P fd65e5165204496787f7e3023d03afce: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260925 02:57:31.434751 27461 tablet_bootstrap.cc:492] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6: Bootstrap starting.
I20260925 02:57:31.436048 27461 tablet_bootstrap.cc:654] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:31.437275 27460 tablet_bootstrap.cc:492] T 0ac1b9484d30407c94fa0886d317d274 P 45a8bfe724a24cadb6ec77a4a749b7ef: Bootstrap starting.
I20260925 02:57:31.439622 27460 tablet_bootstrap.cc:654] T 0ac1b9484d30407c94fa0886d317d274 P 45a8bfe724a24cadb6ec77a4a749b7ef: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:31.440395 27461 tablet_bootstrap.cc:492] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6: No bootstrap required, opened a new log
I20260925 02:57:31.440544 27461 ts_tablet_manager.cc:1397] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260925 02:57:31.441275 27461 raft_consensus.cc:348] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:31.441587 27461 raft_consensus.cc:374] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:31.441702 27461 raft_consensus.cc:729] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a617a0a6960433e9684dd9896366db6, State: Initialized, Role: FOLLOWER
I20260925 02:57:31.441991 27461 consensus_queue.cc:260] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:31.442194 27460 tablet_bootstrap.cc:492] T 0ac1b9484d30407c94fa0886d317d274 P 45a8bfe724a24cadb6ec77a4a749b7ef: No bootstrap required, opened a new log
I20260925 02:57:31.442335 27460 ts_tablet_manager.cc:1397] T 0ac1b9484d30407c94fa0886d317d274 P 45a8bfe724a24cadb6ec77a4a749b7ef: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:57:31.442718 27461 ts_tablet_manager.cc:1428] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 02:57:31.443307 27460 raft_consensus.cc:348] T 0ac1b9484d30407c94fa0886d317d274 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:31.443588 27460 raft_consensus.cc:374] T 0ac1b9484d30407c94fa0886d317d274 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:31.443722 27460 raft_consensus.cc:729] T 0ac1b9484d30407c94fa0886d317d274 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a8bfe724a24cadb6ec77a4a749b7ef, State: Initialized, Role: FOLLOWER
I20260925 02:57:31.443981 27460 consensus_queue.cc:260] T 0ac1b9484d30407c94fa0886d317d274 P 45a8bfe724a24cadb6ec77a4a749b7ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:31.444566 27460 ts_tablet_manager.cc:1428] T 0ac1b9484d30407c94fa0886d317d274 P 45a8bfe724a24cadb6ec77a4a749b7ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:57:31.553714 27387 raft_consensus.cc:482] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:57:31.553920 27387 raft_consensus.cc:504] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:31.555615 27387 leader_election.cc:288] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45a8bfe724a24cadb6ec77a4a749b7ef (127.18.196.193:33577), fd65e5165204496787f7e3023d03afce (127.18.196.194:41945)
I20260925 02:57:31.562691 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:31.556050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:51187 (local address 127.18.196.193:33577)
0925 02:57:31.556244 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:31.556251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:31.556406 (+ 155us) server_negotiation.cc:407] Connection header received
0925 02:57:31.556711 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:31.556716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:31.556809 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:31.556891 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:31.557632 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.558509 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.560785 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.561765 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.561970 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:31.562059 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:31.562285 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 02:57:31.562406 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:57:31.563011 27464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:31.555724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33577 (local address 127.18.196.195:51187)
0925 02:57:31.556309 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:31.556329 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:31.556516 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:31.557115 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:31.557119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:31.557145 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:31.557406 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:31.557417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.558656 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.558662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:31.560585 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:31.560606 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.561955 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.561961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:31.562129 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:31.562152 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:57:31.562258 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 02:57:31.562383 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":24,"thread_start_us":877,"threads_started":1}
I20260925 02:57:31.563665 27377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:31.556403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:45053 (local address 127.18.196.194:41945)
0925 02:57:31.556588 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:31.556596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:31.557402 (+ 806us) server_negotiation.cc:407] Connection header received
0925 02:57:31.557597 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:31.557602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:31.557690 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:31.557779 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:57:31.558533 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.559333 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.561114 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.562396 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.562551 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:31.562652 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:31.563218 (+ 566us) server_negotiation.cc:299] Negotiation successful
0925 02:57:31.563354 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:57:31.563668 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac1b9484d30407c94fa0886d317d274" candidate_uuid: "7a617a0a6960433e9684dd9896366db6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" is_pre_election: true
I20260925 02:57:31.563937 27031 raft_consensus.cc:2454] T 0ac1b9484d30407c94fa0886d317d274 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a617a0a6960433e9684dd9896366db6 in term 0.
I20260925 02:57:31.564368 27465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:31.555852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:41945 (local address 127.18.196.195:45053)
0925 02:57:31.557267 (+ 1415us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:31.557289 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:31.557416 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:31.557944 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:31.557950 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:31.557976 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:57:31.558356 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:31.558367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.559461 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.559467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:31.560934 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:31.560950 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.562800 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.562806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:31.563031 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:31.563067 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 02:57:31.563851 (+ 784us) client_negotiation.cc:240] Negotiation successful
0925 02:57:31.563985 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1270,"mutex_wait_us":41,"thread_start_us":1027,"threads_started":1}
I20260925 02:57:31.564412 27240 leader_election.cc:302] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a8bfe724a24cadb6ec77a4a749b7ef, 7a617a0a6960433e9684dd9896366db6; no voters:
I20260925 02:57:31.564851 27387 raft_consensus.cc:2781] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:57:31.564976 27387 raft_consensus.cc:482] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:57:31.565099 27387 raft_consensus.cc:3037] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:31.565086 27167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac1b9484d30407c94fa0886d317d274" candidate_uuid: "7a617a0a6960433e9684dd9896366db6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd65e5165204496787f7e3023d03afce" is_pre_election: true
I20260925 02:57:31.565336 27167 raft_consensus.cc:2454] T 0ac1b9484d30407c94fa0886d317d274 P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a617a0a6960433e9684dd9896366db6 in term 0.
I20260925 02:57:31.568418 27387 raft_consensus.cc:504] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:31.568996 27387 leader_election.cc:288] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6 [CANDIDATE]: Term 1 election: Requested vote from peers 45a8bfe724a24cadb6ec77a4a749b7ef (127.18.196.193:33577), fd65e5165204496787f7e3023d03afce (127.18.196.194:41945)
I20260925 02:57:31.569422 27167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac1b9484d30407c94fa0886d317d274" candidate_uuid: "7a617a0a6960433e9684dd9896366db6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd65e5165204496787f7e3023d03afce"
I20260925 02:57:31.569409 27031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac1b9484d30407c94fa0886d317d274" candidate_uuid: "7a617a0a6960433e9684dd9896366db6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef"
I20260925 02:57:31.569605 27031 raft_consensus.cc:3037] T 0ac1b9484d30407c94fa0886d317d274 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:31.569613 27167 raft_consensus.cc:3037] T 0ac1b9484d30407c94fa0886d317d274 P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:31.571621 27167 raft_consensus.cc:2454] T 0ac1b9484d30407c94fa0886d317d274 P fd65e5165204496787f7e3023d03afce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a617a0a6960433e9684dd9896366db6 in term 1.
I20260925 02:57:31.571996 27239 leader_election.cc:302] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a617a0a6960433e9684dd9896366db6, fd65e5165204496787f7e3023d03afce; no voters:
I20260925 02:57:31.572256 27387 raft_consensus.cc:2781] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:31.572595 27031 raft_consensus.cc:2454] T 0ac1b9484d30407c94fa0886d317d274 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a617a0a6960433e9684dd9896366db6 in term 1.
I20260925 02:57:31.572706 27387 raft_consensus.cc:686] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6 [term 1 LEADER]: Becoming Leader. State: Replica: 7a617a0a6960433e9684dd9896366db6, State: Running, Role: LEADER
I20260925 02:57:31.573081 27387 consensus_queue.cc:237] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:31.576268 26866 catalog_manager.cc:5495] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a617a0a6960433e9684dd9896366db6 (127.18.196.195). New cstate: current_term: 1 leader_uuid: "7a617a0a6960433e9684dd9896366db6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } health_report { overall_health: HEALTHY } } }
I20260925 02:57:31.613400 27469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:31.608291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41286 (local address 127.18.196.195:45013)
0925 02:57:31.608817 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:31.608821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:31.608839 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:57:31.608913 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:31.608916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:31.608986 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:31.609117 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:57:31.609423 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.610102 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.611372 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.611742 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.611826 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:31.612193 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:57:31.612751 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:57:31.612883 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:31.612960 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 02:57:31.613018 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":309,"threads_started":1}
I20260925 02:57:31.616870 27167 raft_consensus.cc:1261] T 0ac1b9484d30407c94fa0886d317d274 P fd65e5165204496787f7e3023d03afce [term 1 FOLLOWER]: Refusing update from remote peer 7a617a0a6960433e9684dd9896366db6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:57:31.616873 27031 raft_consensus.cc:1261] T 0ac1b9484d30407c94fa0886d317d274 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 1 FOLLOWER]: Refusing update from remote peer 7a617a0a6960433e9684dd9896366db6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:57:31.617642 27387 consensus_queue.cc:1035] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:57:31.617942 27466 consensus_queue.cc:1035] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:57:31.633653 26867 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:57:31.633877 26867 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:57:31.637228 26867 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a: Sending DeleteTablet for 3 replicas of tablet 0ac1b9484d30407c94fa0886d317d274
I20260925 02:57:31.638399 27287 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ac1b9484d30407c94fa0886d317d274 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:31 UTC) from {username='oryx', principal='oryx/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:53990
I20260925 02:57:31.638729 27286 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ac1b9484d30407c94fa0886d317d274 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:31 UTC) from {username='oryx', principal='oryx/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:53990
I20260925 02:57:31.639042 27444 tablet_replica.cc:330] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6: stopping tablet replica
I20260925 02:57:31.638999 26866 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 02:57:31.639253 27444 raft_consensus.cc:2229] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6 [term 1 LEADER]: Raft consensus shutting down.
W20260925 02:57:31.639647 26866 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:57:31.639734 27444 raft_consensus.cc:2258] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:57:31.640064 27006 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ac1b9484d30407c94fa0886d317d274 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:31 UTC) from {username='oryx', principal='oryx/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:54192
I20260925 02:57:31.640363 27442 tablet_replica.cc:330] T 0ac1b9484d30407c94fa0886d317d274 P 45a8bfe724a24cadb6ec77a4a749b7ef: stopping tablet replica
I20260925 02:57:31.640563 27442 raft_consensus.cc:2229] T 0ac1b9484d30407c94fa0886d317d274 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:57:31.640762 27442 raft_consensus.cc:2258] T 0ac1b9484d30407c94fa0886d317d274 P 45a8bfe724a24cadb6ec77a4a749b7ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:57:31.641057 27147 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ac1b9484d30407c94fa0886d317d274 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:31 UTC) from {username='oryx', principal='oryx/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:45178
I20260925 02:57:31.641311 27443 tablet_replica.cc:330] T 0ac1b9484d30407c94fa0886d317d274 P fd65e5165204496787f7e3023d03afce: stopping tablet replica
I20260925 02:57:31.641548 27443 raft_consensus.cc:2229] T 0ac1b9484d30407c94fa0886d317d274 P fd65e5165204496787f7e3023d03afce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:57:31.641784 27443 raft_consensus.cc:2258] T 0ac1b9484d30407c94fa0886d317d274 P fd65e5165204496787f7e3023d03afce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:57:31.642664 27444 ts_tablet_manager.cc:1905] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:57:31.643101 27442 ts_tablet_manager.cc:1905] T 0ac1b9484d30407c94fa0886d317d274 P 45a8bfe724a24cadb6ec77a4a749b7ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:57:31.647413 27443 ts_tablet_manager.cc:1905] T 0ac1b9484d30407c94fa0886d317d274 P fd65e5165204496787f7e3023d03afce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:57:31.647830 27444 ts_tablet_manager.cc:1918] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:57:31.648029 27444 log.cc:1196] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/0ac1b9484d30407c94fa0886d317d274
I20260925 02:57:31.648437 27444 ts_tablet_manager.cc:1939] T 0ac1b9484d30407c94fa0886d317d274 P 7a617a0a6960433e9684dd9896366db6: Deleting consensus metadata
I20260925 02:57:31.649468 26855 catalog_manager.cc:4841] TS 7a617a0a6960433e9684dd9896366db6 (127.18.196.195:45013): tablet 0ac1b9484d30407c94fa0886d317d274 (table test-table [id=51cb89fef9124970aa81c4b29bccbcbc]) successfully deleted
I20260925 02:57:31.649550 27442 ts_tablet_manager.cc:1918] T 0ac1b9484d30407c94fa0886d317d274 P 45a8bfe724a24cadb6ec77a4a749b7ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:57:31.649690 27442 log.cc:1196] T 0ac1b9484d30407c94fa0886d317d274 P 45a8bfe724a24cadb6ec77a4a749b7ef: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-0/wal/wals/0ac1b9484d30407c94fa0886d317d274
I20260925 02:57:31.650100 27442 ts_tablet_manager.cc:1939] T 0ac1b9484d30407c94fa0886d317d274 P 45a8bfe724a24cadb6ec77a4a749b7ef: Deleting consensus metadata
W20260925 02:57:31.650367 26855 catalog_manager.cc:4808] TS 7a617a0a6960433e9684dd9896366db6 (127.18.196.195:45013): delete failed for tablet 0ac1b9484d30407c94fa0886d317d274 because the tablet was not found. No further retry: Not found: Tablet not found: 0ac1b9484d30407c94fa0886d317d274
I20260925 02:57:31.651104 26855 catalog_manager.cc:4841] TS 45a8bfe724a24cadb6ec77a4a749b7ef (127.18.196.193:33577): tablet 0ac1b9484d30407c94fa0886d317d274 (table test-table [id=51cb89fef9124970aa81c4b29bccbcbc]) successfully deleted
I20260925 02:57:31.652779 27443 ts_tablet_manager.cc:1918] T 0ac1b9484d30407c94fa0886d317d274 P fd65e5165204496787f7e3023d03afce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:57:31.652894 27443 log.cc:1196] T 0ac1b9484d30407c94fa0886d317d274 P fd65e5165204496787f7e3023d03afce: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/0ac1b9484d30407c94fa0886d317d274
I20260925 02:57:31.653149 27443 ts_tablet_manager.cc:1939] T 0ac1b9484d30407c94fa0886d317d274 P fd65e5165204496787f7e3023d03afce: Deleting consensus metadata
I20260925 02:57:31.653971 26854 catalog_manager.cc:4841] TS fd65e5165204496787f7e3023d03afce (127.18.196.194:41945): tablet 0ac1b9484d30407c94fa0886d317d274 (table test-table [id=51cb89fef9124970aa81c4b29bccbcbc]) successfully deleted
I20260925 02:57:31.659775 27006 tablet_service.cc:1467] Processing CreateTablet for tablet 6ec25a45cbf44f4898cc702ae9ffe5cb (DEFAULT_TABLE table=test-table [id=80bc2c7f0f4d4a228c131f26c23c0abd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:57:31.660233 27006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ec25a45cbf44f4898cc702ae9ffe5cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:31.660259 27147 tablet_service.cc:1467] Processing CreateTablet for tablet 6ec25a45cbf44f4898cc702ae9ffe5cb (DEFAULT_TABLE table=test-table [id=80bc2c7f0f4d4a228c131f26c23c0abd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:57:31.660670 27147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ec25a45cbf44f4898cc702ae9ffe5cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:31.664548 27460 tablet_bootstrap.cc:492] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef: Bootstrap starting.
I20260925 02:57:31.665234 27286 tablet_service.cc:1467] Processing CreateTablet for tablet 6ec25a45cbf44f4898cc702ae9ffe5cb (DEFAULT_TABLE table=test-table [id=80bc2c7f0f4d4a228c131f26c23c0abd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:57:31.665737 27286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ec25a45cbf44f4898cc702ae9ffe5cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:31.666208 27460 tablet_bootstrap.cc:654] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:31.667309 27458 tablet_bootstrap.cc:492] T 6ec25a45cbf44f4898cc702ae9ffe5cb P fd65e5165204496787f7e3023d03afce: Bootstrap starting.
I20260925 02:57:31.668926 27458 tablet_bootstrap.cc:654] T 6ec25a45cbf44f4898cc702ae9ffe5cb P fd65e5165204496787f7e3023d03afce: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:31.674605 27461 tablet_bootstrap.cc:492] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 7a617a0a6960433e9684dd9896366db6: Bootstrap starting.
I20260925 02:57:31.676904 27461 tablet_bootstrap.cc:654] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 7a617a0a6960433e9684dd9896366db6: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:31.678107 27460 tablet_bootstrap.cc:492] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef: No bootstrap required, opened a new log
I20260925 02:57:31.678311 27460 ts_tablet_manager.cc:1397] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260925 02:57:31.679145 27460 raft_consensus.cc:348] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:31.679602 27460 raft_consensus.cc:374] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:31.679741 27460 raft_consensus.cc:729] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a8bfe724a24cadb6ec77a4a749b7ef, State: Initialized, Role: FOLLOWER
I20260925 02:57:31.680116 27460 consensus_queue.cc:260] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:31.680783 27460 ts_tablet_manager.cc:1428] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:57:31.681459 27458 tablet_bootstrap.cc:492] T 6ec25a45cbf44f4898cc702ae9ffe5cb P fd65e5165204496787f7e3023d03afce: No bootstrap required, opened a new log
I20260925 02:57:31.681630 27458 ts_tablet_manager.cc:1397] T 6ec25a45cbf44f4898cc702ae9ffe5cb P fd65e5165204496787f7e3023d03afce: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20260925 02:57:31.682461 27458 raft_consensus.cc:348] T 6ec25a45cbf44f4898cc702ae9ffe5cb P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:31.682744 27458 raft_consensus.cc:374] T 6ec25a45cbf44f4898cc702ae9ffe5cb P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:31.682847 27458 raft_consensus.cc:729] T 6ec25a45cbf44f4898cc702ae9ffe5cb P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd65e5165204496787f7e3023d03afce, State: Initialized, Role: FOLLOWER
I20260925 02:57:31.683115 27458 consensus_queue.cc:260] T 6ec25a45cbf44f4898cc702ae9ffe5cb P fd65e5165204496787f7e3023d03afce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:31.685158 27461 tablet_bootstrap.cc:492] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 7a617a0a6960433e9684dd9896366db6: No bootstrap required, opened a new log
I20260925 02:57:31.685361 27461 ts_tablet_manager.cc:1397] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 7a617a0a6960433e9684dd9896366db6: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260925 02:57:31.686201 27461 raft_consensus.cc:348] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:31.686376 27458 ts_tablet_manager.cc:1428] T 6ec25a45cbf44f4898cc702ae9ffe5cb P fd65e5165204496787f7e3023d03afce: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260925 02:57:31.686548 27461 raft_consensus.cc:374] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:31.686651 27461 raft_consensus.cc:729] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a617a0a6960433e9684dd9896366db6, State: Initialized, Role: FOLLOWER
I20260925 02:57:31.686983 27461 consensus_queue.cc:260] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 7a617a0a6960433e9684dd9896366db6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:31.687548 27461 ts_tablet_manager.cc:1428] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 7a617a0a6960433e9684dd9896366db6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:57:31.820971 27424 raft_consensus.cc:482] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:57:31.821185 27424 raft_consensus.cc:504] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:31.821949 27424 leader_election.cc:288] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd65e5165204496787f7e3023d03afce (127.18.196.194:41945), 7a617a0a6960433e9684dd9896366db6 (127.18.196.195:45013)
I20260925 02:57:31.822568 27167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec25a45cbf44f4898cc702ae9ffe5cb" candidate_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd65e5165204496787f7e3023d03afce" is_pre_election: true
I20260925 02:57:31.822820 27167 raft_consensus.cc:2454] T 6ec25a45cbf44f4898cc702ae9ffe5cb P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45a8bfe724a24cadb6ec77a4a749b7ef in term 0.
I20260925 02:57:31.823338 26957 leader_election.cc:302] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a8bfe724a24cadb6ec77a4a749b7ef, fd65e5165204496787f7e3023d03afce; no voters:
I20260925 02:57:31.823341 27306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec25a45cbf44f4898cc702ae9ffe5cb" candidate_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a617a0a6960433e9684dd9896366db6" is_pre_election: true
I20260925 02:57:31.823663 27424 raft_consensus.cc:2781] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:57:31.823701 27306 raft_consensus.cc:2454] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45a8bfe724a24cadb6ec77a4a749b7ef in term 0.
I20260925 02:57:31.823769 27424 raft_consensus.cc:482] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:57:31.823889 27424 raft_consensus.cc:3037] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:31.826169 27424 raft_consensus.cc:504] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:31.826793 27424 leader_election.cc:288] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef [CANDIDATE]: Term 1 election: Requested vote from peers fd65e5165204496787f7e3023d03afce (127.18.196.194:41945), 7a617a0a6960433e9684dd9896366db6 (127.18.196.195:45013)
I20260925 02:57:31.827211 27167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec25a45cbf44f4898cc702ae9ffe5cb" candidate_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd65e5165204496787f7e3023d03afce"
I20260925 02:57:31.827421 27167 raft_consensus.cc:3037] T 6ec25a45cbf44f4898cc702ae9ffe5cb P fd65e5165204496787f7e3023d03afce [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:31.829481 27167 raft_consensus.cc:2454] T 6ec25a45cbf44f4898cc702ae9ffe5cb P fd65e5165204496787f7e3023d03afce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45a8bfe724a24cadb6ec77a4a749b7ef in term 1.
I20260925 02:57:31.829922 26957 leader_election.cc:302] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a8bfe724a24cadb6ec77a4a749b7ef, fd65e5165204496787f7e3023d03afce; no voters:
I20260925 02:57:31.829962 27306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec25a45cbf44f4898cc702ae9ffe5cb" candidate_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a617a0a6960433e9684dd9896366db6"
I20260925 02:57:31.830197 27306 raft_consensus.cc:3037] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 7a617a0a6960433e9684dd9896366db6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:31.830262 27424 raft_consensus.cc:2781] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:31.834383 27306 raft_consensus.cc:2454] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 7a617a0a6960433e9684dd9896366db6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45a8bfe724a24cadb6ec77a4a749b7ef in term 1.
I20260925 02:57:31.834905 27424 raft_consensus.cc:686] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef [term 1 LEADER]: Becoming Leader. State: Replica: 45a8bfe724a24cadb6ec77a4a749b7ef, State: Running, Role: LEADER
I20260925 02:57:31.835176 27424 consensus_queue.cc:237] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } }
I20260925 02:57:31.837817 26868 catalog_manager.cc:5495] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 45a8bfe724a24cadb6ec77a4a749b7ef (127.18.196.193). New cstate: current_term: 1 leader_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a8bfe724a24cadb6ec77a4a749b7ef" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 } health_report { overall_health: UNKNOWN } } }
I20260925 02:57:31.888105 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:31.882608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41208 (local address 127.18.196.193:33577)
0925 02:57:31.882776 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:31.882783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:31.882824 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:57:31.883030 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:31.883037 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:31.883128 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:31.883219 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:57:31.883637 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.884778 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.886159 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.886778 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.886952 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:31.887350 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:57:31.887464 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:57:31.887570 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:31.887754 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 02:57:31.887829 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:57:31.900969 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:31.895809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:39681 (local address 127.18.196.193:33577)
0925 02:57:31.896016 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:31.896022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:31.896159 (+ 137us) server_negotiation.cc:407] Connection header received
0925 02:57:31.896355 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:31.896360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:31.896425 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:31.896513 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:57:31.896861 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.897560 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.898968 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.899564 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.899779 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:31.900176 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:57:31.900281 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:57:31.900373 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:31.900554 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 02:57:31.900617 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":64}
I20260925 02:57:31.900993 27483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:31.895561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33577 (local address 127.18.196.193:39681)
0925 02:57:31.896075 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:31.896092 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:31.896249 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:31.896534 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:31.896539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:31.896549 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:57:31.896742 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:31.896750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.897681 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.897688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:31.898783 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:31.898795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:31.899790 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:31.899827 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:31.899989 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:31.900020 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:57:31.900434 (+ 414us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:57:31.900447 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 02:57:31.900535 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 02:57:31.900653 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":474,"threads_started":1}
I20260925 02:57:31.909521 27306 raft_consensus.cc:1261] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 7a617a0a6960433e9684dd9896366db6 [term 1 FOLLOWER]: Refusing update from remote peer 45a8bfe724a24cadb6ec77a4a749b7ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:57:31.909650 27167 raft_consensus.cc:1261] T 6ec25a45cbf44f4898cc702ae9ffe5cb P fd65e5165204496787f7e3023d03afce [term 1 FOLLOWER]: Refusing update from remote peer 45a8bfe724a24cadb6ec77a4a749b7ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:57:31.910109 27424 consensus_queue.cc:1035] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a617a0a6960433e9684dd9896366db6" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 45013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:57:31.910494 27478 consensus_queue.cc:1035] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd65e5165204496787f7e3023d03afce" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 41945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:57:31.992762 26868 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:57:31.992996 26868 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:57:31.996223 26868 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eed677241c274560b0d85c7eabd90e0a: Sending DeleteTablet for 3 replicas of tablet 6ec25a45cbf44f4898cc702ae9ffe5cb
I20260925 02:57:31.997216 27006 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ec25a45cbf44f4898cc702ae9ffe5cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:31 UTC) from {username='oryx', principal='oryx/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:54192
I20260925 02:57:31.997517 27442 tablet_replica.cc:330] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef: stopping tablet replica
I20260925 02:57:31.997711 27442 raft_consensus.cc:2229] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:57:31.997752 27286 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ec25a45cbf44f4898cc702ae9ffe5cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:31 UTC) from {username='oryx', principal='oryx/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:53990
I20260925 02:57:31.998061 27147 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ec25a45cbf44f4898cc702ae9ffe5cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:57:31 UTC) from {username='oryx', principal='oryx/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:45178
I20260925 02:57:31.998207 27442 raft_consensus.cc:2258] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:57:31.998375 27444 tablet_replica.cc:330] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 7a617a0a6960433e9684dd9896366db6: stopping tablet replica
I20260925 02:57:31.998555 27444 raft_consensus.cc:2229] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 7a617a0a6960433e9684dd9896366db6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:57:31.998737 27444 raft_consensus.cc:2258] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 7a617a0a6960433e9684dd9896366db6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:57:31.999030 27443 tablet_replica.cc:330] T 6ec25a45cbf44f4898cc702ae9ffe5cb P fd65e5165204496787f7e3023d03afce: stopping tablet replica
I20260925 02:57:31.999193 27443 raft_consensus.cc:2229] T 6ec25a45cbf44f4898cc702ae9ffe5cb P fd65e5165204496787f7e3023d03afce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:57:31.999374 27443 raft_consensus.cc:2258] T 6ec25a45cbf44f4898cc702ae9ffe5cb P fd65e5165204496787f7e3023d03afce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:57:32.000391 27442 ts_tablet_manager.cc:1905] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:57:32.001060 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 26939
I20260925 02:57:32.004818 27444 ts_tablet_manager.cc:1905] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 7a617a0a6960433e9684dd9896366db6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:57:32.007596 27442 ts_tablet_manager.cc:1918] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:57:32.007737 27442 log.cc:1196] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-0/wal/wals/6ec25a45cbf44f4898cc702ae9ffe5cb
I20260925 02:57:32.008203 27442 ts_tablet_manager.cc:1939] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 45a8bfe724a24cadb6ec77a4a749b7ef: Deleting consensus metadata
I20260925 02:57:32.012745 27444 ts_tablet_manager.cc:1918] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 7a617a0a6960433e9684dd9896366db6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:57:32.012871 27444 log.cc:1196] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 7a617a0a6960433e9684dd9896366db6: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/6ec25a45cbf44f4898cc702ae9ffe5cb
I20260925 02:57:32.013250 27444 ts_tablet_manager.cc:1939] T 6ec25a45cbf44f4898cc702ae9ffe5cb P 7a617a0a6960433e9684dd9896366db6: Deleting consensus metadata
I20260925 02:57:32.014312 26855 catalog_manager.cc:4841] TS 7a617a0a6960433e9684dd9896366db6 (127.18.196.195:45013): tablet 6ec25a45cbf44f4898cc702ae9ffe5cb (table test-table [id=80bc2c7f0f4d4a228c131f26c23c0abd]) successfully deleted
I20260925 02:57:32.015198 26855 catalog_manager.cc:4841] TS 45a8bfe724a24cadb6ec77a4a749b7ef (127.18.196.193:33577): tablet 6ec25a45cbf44f4898cc702ae9ffe5cb (table test-table [id=80bc2c7f0f4d4a228c131f26c23c0abd]) successfully deleted
I20260925 02:57:32.018698 27443 ts_tablet_manager.cc:1905] T 6ec25a45cbf44f4898cc702ae9ffe5cb P fd65e5165204496787f7e3023d03afce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:57:32.025813 27443 ts_tablet_manager.cc:1918] T 6ec25a45cbf44f4898cc702ae9ffe5cb P fd65e5165204496787f7e3023d03afce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:57:32.026436 27443 log.cc:1196] T 6ec25a45cbf44f4898cc702ae9ffe5cb P fd65e5165204496787f7e3023d03afce: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/6ec25a45cbf44f4898cc702ae9ffe5cb
I20260925 02:57:32.027333 27443 ts_tablet_manager.cc:1939] T 6ec25a45cbf44f4898cc702ae9ffe5cb P fd65e5165204496787f7e3023d03afce: Deleting consensus metadata
I20260925 02:57:32.029562 26854 catalog_manager.cc:4841] TS fd65e5165204496787f7e3023d03afce (127.18.196.194:41945): tablet 6ec25a45cbf44f4898cc702ae9ffe5cb (table test-table [id=80bc2c7f0f4d4a228c131f26c23c0abd]) successfully deleted
Sep 25 02:57:32 dist-test-slave-8zvk krb5kdc[26823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.196.193@KRBTEST.COM
I20260925 02:57:32.051988 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:32.014690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41218 (local address 127.18.196.193:33577)
0925 02:57:32.015012 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:32.015018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:32.032665 (+ 17647us) server_negotiation.cc:407] Connection header received
0925 02:57:32.033142 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:32.033147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:32.033253 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:32.033335 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:32.035139 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:32.036222 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:32.038098 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:32.038786 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:32.038923 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:32.044297 (+ 5374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:32.044320 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:32.044340 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:32.044396 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:32.048490 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:32.049351 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:32.049360 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:32.049363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:32.049554 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:32.050030 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:32.050035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:32.050039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:32.050474 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:32.050786 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:32.051415 (+ 629us) server_negotiation.cc:299] Negotiation successful
0925 02:57:32.051630 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212}
I20260925 02:57:32.053373 27072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:32.186648 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 26939
I20260925 02:57:32.210820 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 27081
Sep 25 02:57:32 dist-test-slave-8zvk krb5kdc[26823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.196.194@KRBTEST.COM
I20260925 02:57:32.225814 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:32.211631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51092 (local address 127.18.196.194:41945)
0925 02:57:32.212090 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:32.212095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:32.212111 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:57:32.212169 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:32.212173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:32.212240 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:32.212333 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:32.213586 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:32.214304 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:32.215541 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:32.215963 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:32.216066 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:32.220120 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:32.220131 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:32.220144 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:32.220175 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:32.222797 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:32.223765 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:32.223771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:32.223773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:32.223939 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:32.224353 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:32.224357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:32.224359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:32.224760 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:32.224986 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:32.225280 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 02:57:32.225487 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":257,"threads_started":1}
I20260925 02:57:32.226440 27212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:32.351131 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 27081
I20260925 02:57:32.374665 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 27221
Sep 25 02:57:32 dist-test-slave-8zvk krb5kdc[26823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.196.195@KRBTEST.COM
I20260925 02:57:32.395084 27492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:32.375480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41288 (local address 127.18.196.195:45013)
0925 02:57:32.375978 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:32.375983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:32.375998 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:57:32.376063 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:32.376068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:32.376125 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:32.376215 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:32.377551 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:32.378310 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:32.379693 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:32.380136 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:32.380247 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:32.389797 (+ 9550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:32.389813 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:32.389830 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:32.389892 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:32.392621 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:32.393288 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:32.393294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:32.393297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:32.393423 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:32.393750 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:32.393753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:32.393755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:32.394119 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:32.394346 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:32.394601 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 02:57:32.394770 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":288,"threads_started":1}
I20260925 02:57:32.395720 27352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:32.520717 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 27221
I20260925 02:57:32.545899 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 26836
I20260925 02:57:32.547097 26908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:32.663640 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 26836
2026-09-25T02:57:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5505 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:57:32 dist-test-slave-8zvk krb5kdc[27497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:57:32 dist-test-slave-8zvk krb5kdc[27497](info): set up 2 sockets
Sep 25 02:57:32 dist-test-slave-8zvk krb5kdc[27497](info): commencing operation
krb5kdc: starting...
W20260925 02:57:33.751075 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 02:57:33 dist-test-slave-8zvk krb5kdc[27497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:57:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:57:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 02:57:33.974521 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:44043
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:42345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:44043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:34.294322 27510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:34.294677 27510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:34.294889 27510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:34.302217 27510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:34.302367 27510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:34.302488 27510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:34.302619 27510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:34.302686 27510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:34.312000 27510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:42345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:44043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:44043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:34.313189 27510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:34.314417 27510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:34.325632 27515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:34.325603 27518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:34.325693 27516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:34.326718 27510 server_base.cc:1034] running on GCE node
I20260925 02:57:34.327477 27510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:34.328732 27510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:34.329918 27510 hybrid_clock.cc:648] HybridClock initialized: now 1790305054329860 us; error 44 us; skew 500 ppm
I20260925 02:57:34.330247 27510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T02:57:34Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1726 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:57:34 dist-test-slave-8zvk krb5kdc[27524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:57:34 dist-test-slave-8zvk krb5kdc[27524](info): set up 2 sockets
Sep 25 02:57:34 dist-test-slave-8zvk krb5kdc[27524](info): commencing operation
krb5kdc: starting...
W20260925 02:57:35.479007 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 02:57:35 dist-test-slave-8zvk krb5kdc[27524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:57:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:57:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 02:57:35.695493 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:40393
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:40393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:36.012137 27537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:36.012396 27537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:36.012475 27537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:36.019393 27537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:36.019485 27537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:36.019528 27537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:36.019591 27537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:36.019627 27537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:36.028908 27537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:40393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:40393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:36.029953 27537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:36.031082 27537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:36.042963 27542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:36.042909 27545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:36.042975 27543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:36.044108 27537 server_base.cc:1034] running on GCE node
I20260925 02:57:36.044847 27537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:36.046118 27537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:36.047349 27537 hybrid_clock.cc:648] HybridClock initialized: now 1790305056047308 us; error 44 us; skew 500 ppm
I20260925 02:57:36.047693 27537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:36 dist-test-slave-8zvk krb5kdc[27524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305056, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:36.050640 27537 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 02:57:36.052155 27537 webserver.cc:460] Webserver started at http://127.18.196.254:42919/ using document root <none> and password file <none>
I20260925 02:57:36.052816 27537 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:36.052963 27537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:36.053244 27537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:36.056109 27537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "73425aedcb934f0c8b080a5c681a93fa"
format_stamp: "Formatted at 2026-09-25 02:57:36 on dist-test-slave-8zvk"
server_key: "f2312bb48a73560e58291b563b87b08b"
server_key_iv: "56aebdb60400a9d228407805beb824a2"
server_key_version: "encryptionkey@0"
I20260925 02:57:36.056861 27537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "73425aedcb934f0c8b080a5c681a93fa"
format_stamp: "Formatted at 2026-09-25 02:57:36 on dist-test-slave-8zvk"
server_key: "f2312bb48a73560e58291b563b87b08b"
server_key_iv: "56aebdb60400a9d228407805beb824a2"
server_key_version: "encryptionkey@0"
I20260925 02:57:36.061592 27537 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 02:57:36.064831 27553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:36.066057 27537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:57:36.066236 27537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "73425aedcb934f0c8b080a5c681a93fa"
format_stamp: "Formatted at 2026-09-25 02:57:36 on dist-test-slave-8zvk"
server_key: "f2312bb48a73560e58291b563b87b08b"
server_key_iv: "56aebdb60400a9d228407805beb824a2"
server_key_version: "encryptionkey@0"
I20260925 02:57:36.066452 27537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:36.084384 27537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:36.088074 27537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:36.088338 27537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:36.106945 27537 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:40393
I20260925 02:57:36.106949 27604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:40393 every 8 connection(s)
I20260925 02:57:36.108306 27537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:57:36.111753 27605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:36.115329 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 27537
I20260925 02:57:36.115528 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:57:36.116132 19219 external_mini_cluster.cc:1428] Setting key d81b019ea0597c247203317c11ad9aa1
I20260925 02:57:36.123385 27605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa: Bootstrap starting.
I20260925 02:57:36.126781 27605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:36.127869 27605 log.cc:824] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:36.130425 27605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa: No bootstrap required, opened a new log
Sep 25 02:57:36 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:57:36.137586 27605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73425aedcb934f0c8b080a5c681a93fa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40393 } }
I20260925 02:57:36.137945 27605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:36.138077 27605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73425aedcb934f0c8b080a5c681a93fa, State: Initialized, Role: FOLLOWER
I20260925 02:57:36.138679 27605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73425aedcb934f0c8b080a5c681a93fa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40393 } }
I20260925 02:57:36.138963 27605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:57:36.139115 27605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:57:36.139294 27605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:36.139642 27608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:36.118027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45156 (local address 127.18.196.254:40393)
0925 02:57:36.118791 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:36.118808 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:36.118885 (+ 77us) server_negotiation.cc:407] Connection header received
0925 02:57:36.121776 (+ 2891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:36.121803 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:36.122255 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:36.122671 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:36.124189 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:36.126154 (+ 1965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:36.127540 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:36.128605 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:36.128839 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:36.132576 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:36.132584 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:36.132603 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:36.132641 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:36.135628 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:36.136268 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:36.136274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:36.136280 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:36.136466 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:36.136868 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:36.136872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:36.136874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:36.137330 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:36.137629 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:36.138050 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 02:57:36.138486 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":306,"threads_started":1}
I20260925 02:57:36.142043 27605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73425aedcb934f0c8b080a5c681a93fa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40393 } }
I20260925 02:57:36.142618 27605 leader_election.cc:302] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73425aedcb934f0c8b080a5c681a93fa; no voters:
I20260925 02:57:36.143182 27605 leader_election.cc:288] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:57:36.143275 27610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:36.144663 27610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [term 1 LEADER]: Becoming Leader. State: Replica: 73425aedcb934f0c8b080a5c681a93fa, State: Running, Role: LEADER
I20260925 02:57:36.145082 27610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73425aedcb934f0c8b080a5c681a93fa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40393 } }
I20260925 02:57:36.145190 27605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:36.147353 27612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73425aedcb934f0c8b080a5c681a93fa. Latest consensus state: current_term: 1 leader_uuid: "73425aedcb934f0c8b080a5c681a93fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73425aedcb934f0c8b080a5c681a93fa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40393 } } }
I20260925 02:57:36.147660 27612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:36.148262 27611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73425aedcb934f0c8b080a5c681a93fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73425aedcb934f0c8b080a5c681a93fa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40393 } } }
I20260925 02:57:36.148537 27611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:36.148536 27617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:36.153105 27617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:36.161458 27617 catalog_manager.cc:1347] Generated new cluster ID: 5209ab80f45d446986f6e765d004a227
I20260925 02:57:36.161571 27617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:36.173348 27617 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:57:36.175194 27617 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:36.183351 27617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa: Generated new TSK 0
I20260925 02:57:36.184204 27617 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
I20260925 02:57:36.246662 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:40393
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:36.568387 27631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:36.568683 27631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:36.568836 27631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:36.575780 27631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:36.575917 27631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:36.576093 27631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 02:57:36.585260 27631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:40393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:36.586522 27631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:36.587798 27631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:36.600651 27636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:36.600737 27637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:36.600694 27639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:36.601583 27631 server_base.cc:1034] running on GCE node
I20260925 02:57:36.602195 27631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:36.603117 27631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:36.604337 27631 hybrid_clock.cc:648] HybridClock initialized: now 1790305056604290 us; error 35 us; skew 500 ppm
I20260925 02:57:36.604665 27631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:36 dist-test-slave-8zvk krb5kdc[27524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305056, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:36.607795 27631 init.cc:377] Logged in from keytab as kudu/127.18.196.193@KRBTEST.COM (short username kudu)
I20260925 02:57:36.609200 27631 webserver.cc:460] Webserver started at http://127.18.196.193:40987/ using document root <none> and password file <none>
I20260925 02:57:36.609863 27631 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:36.610013 27631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:36.610339 27631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:36.613076 27631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "2a8ebe349b0a4756a17dca196fa89659"
format_stamp: "Formatted at 2026-09-25 02:57:36 on dist-test-slave-8zvk"
server_key: "4481acc42473689737a6d2c295553a3c"
server_key_iv: "0dabd3fd978acd627959547c8c79af31"
server_key_version: "encryptionkey@0"
I20260925 02:57:36.613821 27631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "2a8ebe349b0a4756a17dca196fa89659"
format_stamp: "Formatted at 2026-09-25 02:57:36 on dist-test-slave-8zvk"
server_key: "4481acc42473689737a6d2c295553a3c"
server_key_iv: "0dabd3fd978acd627959547c8c79af31"
server_key_version: "encryptionkey@0"
I20260925 02:57:36.618551 27631 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260925 02:57:36.621833 27646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:36.623172 27631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:57:36.623342 27631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "2a8ebe349b0a4756a17dca196fa89659"
format_stamp: "Formatted at 2026-09-25 02:57:36 on dist-test-slave-8zvk"
server_key: "4481acc42473689737a6d2c295553a3c"
server_key_iv: "0dabd3fd978acd627959547c8c79af31"
server_key_version: "encryptionkey@0"
I20260925 02:57:36.623538 27631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:36.654318 27631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:36.658581 27631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:36.658924 27631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:36.659843 27631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:36.661110 27631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:36.661211 27631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:36.661352 27631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:36.661460 27631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:36.694074 27631 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:40783
I20260925 02:57:36.694129 27759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:40783 every 8 connection(s)
I20260925 02:57:36.695402 27631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 02:57:36.699648 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 27631
I20260925 02:57:36.699847 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 02:57:36.700464 19219 external_mini_cluster.cc:1428] Setting key 6eab86ee0e5942bd1d8cf8e8bf7f1016
Sep 25 02:57:36 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305056, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:57:36.714485 27762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:36.697877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:48273 (local address 127.18.196.254:40393)
0925 02:57:36.698307 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:36.698312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:36.699253 (+ 941us) server_negotiation.cc:407] Connection header received
0925 02:57:36.701257 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:36.701263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:36.701348 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:36.701487 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:36.703275 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:36.703961 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:36.705458 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:36.705849 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:36.705981 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:36.709379 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:36.709384 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:36.709387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:36.709417 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:36.711807 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:36.712471 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:36.712476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:36.712478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:36.712586 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:36.712971 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:36.712975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:36.712978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:36.713219 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:36.713362 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:36.713952 (+ 590us) server_negotiation.cc:299] Negotiation successful
0925 02:57:36.714139 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":216,"threads_started":1}
I20260925 02:57:36.715411 27763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:36.698399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.193:48273)
0925 02:57:36.699101 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:36.699142 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:36.700968 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:36.701761 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:36.701772 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:36.702498 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:36.703113 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:36.703131 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:36.704130 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:36.704134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:36.705307 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:36.705318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:36.706062 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:36.706069 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:36.706315 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:36.707039 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:36.707061 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:36.709151 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:36.711938 (+ 2787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:36.711947 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:36.711960 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:36.712351 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:36.712686 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:36.712690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:36.712694 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:36.712860 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:36.713335 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:36.713342 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:36.713638 (+ 296us) client_negotiation.cc:769] Sending connection context
0925 02:57:36.713858 (+ 220us) client_negotiation.cc:240] Negotiation successful
0925 02:57:36.714320 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":268,"threads_started":1}
I20260925 02:57:36.716908 27760 heartbeater.cc:347] Connected to a master server at 127.18.196.254:40393
I20260925 02:57:36.717176 27760 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:36.717849 27760 heartbeater.cc:511] Master 127.18.196.254:40393 requested a full tablet report, sending...
I20260925 02:57:36.719918 27570 ts_manager.cc:194] Registered new tserver with Master: 2a8ebe349b0a4756a17dca196fa89659 (127.18.196.193:40783)
WARNING: no policy specified for kudu/127.18.196.194@KRBTEST.COM; defaulting to no policy
I20260925 02:57:36.721660 27570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.193@KRBTEST.COM'} at 127.18.196.193:48273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
I20260925 02:57:36.746383 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:40393
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:37.058113 27766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:37.058362 27766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:37.058450 27766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:37.065225 27766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:37.065316 27766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:37.065412 27766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 02:57:37.074590 27766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:40393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:37.075752 27766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:37.076949 27766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:37.089439 27772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:37.089466 27774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:37.089510 27771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:37.090251 27766 server_base.cc:1034] running on GCE node
I20260925 02:57:37.090886 27766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:37.091853 27766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:37.093166 27766 hybrid_clock.cc:648] HybridClock initialized: now 1790305057093100 us; error 58 us; skew 500 ppm
I20260925 02:57:37.093500 27766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:37 dist-test-slave-8zvk krb5kdc[27524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305057, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:37.096678 27766 init.cc:377] Logged in from keytab as kudu/127.18.196.194@KRBTEST.COM (short username kudu)
I20260925 02:57:37.098080 27766 webserver.cc:460] Webserver started at http://127.18.196.194:41389/ using document root <none> and password file <none>
I20260925 02:57:37.098758 27766 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:37.098924 27766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:37.099236 27766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:37.102074 27766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "e8134832c481409aa563523bd1e8190e"
format_stamp: "Formatted at 2026-09-25 02:57:37 on dist-test-slave-8zvk"
server_key: "af0130634424f485be7ffc7f1e24def6"
server_key_iv: "0deafcd3af5d2dc99a6a8ce20e0bb8af"
server_key_version: "encryptionkey@0"
I20260925 02:57:37.102936 27766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "e8134832c481409aa563523bd1e8190e"
format_stamp: "Formatted at 2026-09-25 02:57:37 on dist-test-slave-8zvk"
server_key: "af0130634424f485be7ffc7f1e24def6"
server_key_iv: "0deafcd3af5d2dc99a6a8ce20e0bb8af"
server_key_version: "encryptionkey@0"
I20260925 02:57:37.108227 27766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 02:57:37.112365 27782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:37.113821 27766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:57:37.114005 27766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "e8134832c481409aa563523bd1e8190e"
format_stamp: "Formatted at 2026-09-25 02:57:37 on dist-test-slave-8zvk"
server_key: "af0130634424f485be7ffc7f1e24def6"
server_key_iv: "0deafcd3af5d2dc99a6a8ce20e0bb8af"
server_key_version: "encryptionkey@0"
I20260925 02:57:37.114208 27766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:37.128587 27766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:37.132323 27766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:37.132591 27766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:37.133483 27766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:37.134682 27766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:37.134783 27766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:37.134948 27766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:37.135021 27766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:37.167289 27766 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:33749
I20260925 02:57:37.167335 27895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:33749 every 8 connection(s)
I20260925 02:57:37.168619 27766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 02:57:37.174237 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 27766
I20260925 02:57:37.174401 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 02:57:37.174978 19219 external_mini_cluster.cc:1428] Setting key 852b1a496e0edeaf9455d655340ef4dc
Sep 25 02:57:37 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305057, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:57:37.188931 27762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:37.171402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:34933 (local address 127.18.196.254:40393)
0925 02:57:37.171608 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:37.171614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:37.172802 (+ 1188us) server_negotiation.cc:407] Connection header received
0925 02:57:37.174624 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:37.174630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:37.174710 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:37.174811 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:37.176414 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:37.177384 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:37.178908 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:37.179631 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:37.179760 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:37.182959 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:37.182968 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:37.182975 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:37.183019 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:37.185788 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:37.186447 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:37.186454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:37.186461 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:37.186636 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:37.187108 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:37.187116 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:37.187122 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:37.187505 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:37.187655 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:37.188266 (+ 611us) server_negotiation.cc:299] Negotiation successful
0925 02:57:37.188513 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:57:37.189544 27898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:37.171965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.194:34933)
0925 02:57:37.172645 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:37.172691 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:37.174279 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:37.175088 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:37.175101 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:37.175689 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:37.176255 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:37.176272 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:37.177536 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:37.177540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:37.178719 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:37.178731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:37.179832 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:37.179837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:37.180037 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:37.180683 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:37.180704 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:37.182718 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:37.185908 (+ 3190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:37.185914 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:37.185924 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:37.186326 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:37.186753 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:37.186756 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:37.186758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:37.186996 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:37.187628 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:37.187633 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:37.187902 (+ 269us) client_negotiation.cc:769] Sending connection context
0925 02:57:37.188128 (+ 226us) client_negotiation.cc:240] Negotiation successful
0925 02:57:37.188470 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":275,"threads_started":1}
I20260925 02:57:37.191042 27896 heartbeater.cc:347] Connected to a master server at 127.18.196.254:40393
I20260925 02:57:37.191354 27896 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:37.192051 27896 heartbeater.cc:511] Master 127.18.196.254:40393 requested a full tablet report, sending...
I20260925 02:57:37.193562 27570 ts_manager.cc:194] Registered new tserver with Master: e8134832c481409aa563523bd1e8190e (127.18.196.194:33749)
I20260925 02:57:37.194825 27570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.194@KRBTEST.COM'} at 127.18.196.194:34933
WARNING: no policy specified for kudu/127.18.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
I20260925 02:57:37.230314 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:40393
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:37.542438 27901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:37.542714 27901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:37.542903 27901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:37.549901 27901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:37.550014 27901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:37.550189 27901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 02:57:37.559518 27901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:40393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:37.560619 27901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:37.561888 27901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:37.574757 27907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:37.574786 27906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:37.575107 27909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:37.575382 27901 server_base.cc:1034] running on GCE node
I20260925 02:57:37.575909 27901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:37.576752 27901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:37.577970 27901 hybrid_clock.cc:648] HybridClock initialized: now 1790305057577935 us; error 48 us; skew 500 ppm
I20260925 02:57:37.578254 27901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:37 dist-test-slave-8zvk krb5kdc[27524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305057, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:37.584736 27901 init.cc:377] Logged in from keytab as kudu/127.18.196.195@KRBTEST.COM (short username kudu)
I20260925 02:57:37.585978 27901 webserver.cc:460] Webserver started at http://127.18.196.195:34133/ using document root <none> and password file <none>
I20260925 02:57:37.586632 27901 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:37.586757 27901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:37.587097 27901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:37.589902 27901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "008e3dcb93d344a0b4165607cb6ab7db"
format_stamp: "Formatted at 2026-09-25 02:57:37 on dist-test-slave-8zvk"
server_key: "f3f0d00cf663f197d3381cc9d67a5300"
server_key_iv: "7537fd188ced8908f5a8e55aa33db527"
server_key_version: "encryptionkey@0"
I20260925 02:57:37.590651 27901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "008e3dcb93d344a0b4165607cb6ab7db"
format_stamp: "Formatted at 2026-09-25 02:57:37 on dist-test-slave-8zvk"
server_key: "f3f0d00cf663f197d3381cc9d67a5300"
server_key_iv: "7537fd188ced8908f5a8e55aa33db527"
server_key_version: "encryptionkey@0"
I20260925 02:57:37.595903 27901 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 02:57:37.599252 27916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:37.600494 27901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 02:57:37.600668 27901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "008e3dcb93d344a0b4165607cb6ab7db"
format_stamp: "Formatted at 2026-09-25 02:57:37 on dist-test-slave-8zvk"
server_key: "f3f0d00cf663f197d3381cc9d67a5300"
server_key_iv: "7537fd188ced8908f5a8e55aa33db527"
server_key_version: "encryptionkey@0"
I20260925 02:57:37.600871 27901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:37.614185 27901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:37.618098 27901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:37.618351 27901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:37.619174 27901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:37.620286 27901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:37.620383 27901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:37.620527 27901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:37.620603 27901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:37.652468 27901 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:43967
I20260925 02:57:37.652547 28029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:43967 every 8 connection(s)
I20260925 02:57:37.653683 27901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 02:57:37.660668 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 27901
I20260925 02:57:37.660852 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 02:57:37.661334 19219 external_mini_cluster.cc:1428] Setting key d9dafa26dc49dbbdf91236e3fc50792a
Sep 25 02:57:37 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305057, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:57:37.672574 27762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:37.656170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:37163 (local address 127.18.196.254:40393)
0925 02:57:37.656356 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:37.656360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:37.657458 (+ 1098us) server_negotiation.cc:407] Connection header received
0925 02:57:37.659220 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:37.659224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:37.659315 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:37.659399 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:37.661049 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:37.661845 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:37.663578 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:37.664048 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:37.664190 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:37.667205 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:37.667210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:37.667213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:37.667243 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:37.670009 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:37.670791 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:37.670794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:37.670796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:37.670958 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:37.671299 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:37.671302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:37.671304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:37.671537 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:37.671644 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:37.672189 (+ 545us) server_negotiation.cc:299] Negotiation successful
0925 02:57:37.672349 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:57:37.673405 28032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:37.656593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.195:37163)
0925 02:57:37.657300 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:37.657335 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:37.658988 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:37.659605 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:37.659613 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:37.660099 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:37.660886 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:37.660912 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:37.661992 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:37.661997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:37.663452 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:37.663465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:37.664280 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:37.664289 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:37.664520 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:37.665055 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:37.665074 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:37.667035 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:37.670123 (+ 3088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:37.670145 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:37.670158 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:37.670688 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:37.671045 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:37.671048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:37.671050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:37.671209 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:37.671632 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:37.671638 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:37.671891 (+ 253us) client_negotiation.cc:769] Sending connection context
0925 02:57:37.672089 (+ 198us) client_negotiation.cc:240] Negotiation successful
0925 02:57:37.672425 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":293,"threads_started":1}
I20260925 02:57:37.674731 28030 heartbeater.cc:347] Connected to a master server at 127.18.196.254:40393
I20260925 02:57:37.675092 28030 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:37.675951 28030 heartbeater.cc:511] Master 127.18.196.254:40393 requested a full tablet report, sending...
I20260925 02:57:37.677363 27570 ts_manager.cc:194] Registered new tserver with Master: 008e3dcb93d344a0b4165607cb6ab7db (127.18.196.195:43967)
I20260925 02:57:37.678089 27570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.195@KRBTEST.COM'} at 127.18.196.195:37163
I20260925 02:57:37.688346 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.18.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.196.193@KRBTEST.COM" created.
I20260925 02:57:37.724337 27760 heartbeater.cc:503] Master 127.18.196.254:40393 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.193.keytab.
Entry for principal oryx/127.18.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.193.keytab.
WARNING: no policy specified for oryx/127.18.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.194.keytab.
Entry for principal oryx/127.18.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.194.keytab.
WARNING: no policy specified for oryx/127.18.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.195.keytab.
Entry for principal oryx/127.18.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.195.keytab.
I20260925 02:57:37.817521 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 27631
Sep 25 02:57:37 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.193@KRBTEST.COM
I20260925 02:57:37.832880 28040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:37.818565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50824 (local address 127.18.196.193:40783)
0925 02:57:37.819222 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:37.819229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:37.819258 (+ 29us) server_negotiation.cc:407] Connection header received
0925 02:57:37.819322 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:37.819328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:37.819538 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:37.819673 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:37.820899 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:37.821911 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:37.823316 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:37.823874 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:37.823983 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:37.827957 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:37.827967 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:37.827979 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:37.828016 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:37.830552 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:37.831246 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:37.831252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:37.831255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:37.831371 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:37.831679 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:37.831682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:37.831684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:37.831999 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:37.832207 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:37.832412 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 02:57:37.832596 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":338,"threads_started":1}
I20260925 02:57:37.834019 27755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:37.918020 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 27631
I20260925 02:57:37.934691 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 27766
Sep 25 02:57:37 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.194@KRBTEST.COM
I20260925 02:57:37.951797 28042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:37.935667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36950 (local address 127.18.196.194:33749)
0925 02:57:37.936220 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:37.936231 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:37.936257 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:57:37.936320 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:37.936326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:37.936539 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:37.936665 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:37.937842 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:37.938998 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:37.940209 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:37.940740 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:37.940854 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:37.945054 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:37.945064 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:37.945078 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:37.945167 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:37.948637 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:37.949413 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:37.949422 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:37.949426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:37.949574 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:37.949966 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:37.949971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:37.949974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:37.950360 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:37.950672 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:37.951018 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 02:57:37.951299 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":289,"threads_started":1}
I20260925 02:57:37.953240 27891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:38.034479 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 27766
I20260925 02:57:38.051075 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 27901
Sep 25 02:57:38 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.195@KRBTEST.COM
I20260925 02:57:38.067375 28044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:38.052184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56934 (local address 127.18.196.195:43967)
0925 02:57:38.052789 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:38.052796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:38.052824 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:57:38.052886 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:38.052892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:38.053096 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:38.053215 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:38.054496 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:38.055738 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:38.056910 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:38.057433 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:38.057527 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:38.061778 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:38.061788 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:38.061801 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:38.061832 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:38.064540 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:38.065220 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:38.065227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:38.065229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:38.065360 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:38.065763 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:38.065769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:38.065774 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:38.066101 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:38.066436 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:38.066740 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 02:57:38.067032 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":306,"threads_started":1}
I20260925 02:57:38.068620 28025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:38.150629 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 27901
I20260925 02:57:38.167018 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 27537
I20260925 02:57:38.168180 27600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:38.267890 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 27537
I20260925 02:57:38.288470 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:40393
--webserver_interface=127.18.196.254
--webserver_port=42919
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:40393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:38.599388 28047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:38.599699 28047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:38.599864 28047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:38.607111 28047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:38.607226 28047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:38.607307 28047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:38.607402 28047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:38.607484 28047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:38.617015 28047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:40393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:40393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=42919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:38.618233 28047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:38.619473 28047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:38.630945 28052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:38.630869 28055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:38.630986 28053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:38.631917 28047 server_base.cc:1034] running on GCE node
I20260925 02:57:38.632681 28047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:38.634114 28047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:38.635334 28047 hybrid_clock.cc:648] HybridClock initialized: now 1790305058635284 us; error 62 us; skew 500 ppm
I20260925 02:57:38.635663 28047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:38 dist-test-slave-8zvk krb5kdc[27524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305058, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:38.638798 28047 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 02:57:38.641507 28047 webserver.cc:460] Webserver started at http://127.18.196.254:42919/ using document root <none> and password file <none>
I20260925 02:57:38.642192 28047 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:38.642310 28047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:38.647996 28047 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 02:57:38.650755 28062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:38.652102 28047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:57:38.652287 28047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "73425aedcb934f0c8b080a5c681a93fa"
format_stamp: "Formatted at 2026-09-25 02:57:36 on dist-test-slave-8zvk"
server_key: "f2312bb48a73560e58291b563b87b08b"
server_key_iv: "56aebdb60400a9d228407805beb824a2"
server_key_version: "encryptionkey@0"
I20260925 02:57:38.653288 28047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:38.677134 28047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:38.681116 28047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:38.681403 28047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:38.700064 28047 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:40393
I20260925 02:57:38.700078 28113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:40393 every 8 connection(s)
I20260925 02:57:38.701550 28047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:57:38.708109 28114 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:57:38.708817 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 28047
I20260925 02:57:38.709786 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:40783
--local_ip_for_outbound_sockets=127.18.196.193
--tserver_master_addrs=127.18.196.254:40393
--webserver_port=40987
--webserver_interface=127.18.196.193
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.196.193
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:57:38.713027 28114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa: Bootstrap starting.
I20260925 02:57:38.733172 28114 log.cc:824] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:38.742067 28114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:57:38.742602 28114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa: Bootstrap complete.
I20260925 02:57:38.751727 28114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73425aedcb934f0c8b080a5c681a93fa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40393 } }
I20260925 02:57:38.752523 28114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73425aedcb934f0c8b080a5c681a93fa, State: Initialized, Role: FOLLOWER
I20260925 02:57:38.753170 28114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73425aedcb934f0c8b080a5c681a93fa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40393 } }
I20260925 02:57:38.753412 28114 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:57:38.753527 28114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:57:38.753726 28114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:57:38.757715 28114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73425aedcb934f0c8b080a5c681a93fa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40393 } }
I20260925 02:57:38.758220 28114 leader_election.cc:302] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73425aedcb934f0c8b080a5c681a93fa; no voters:
I20260925 02:57:38.758870 28114 leader_election.cc:288] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:57:38.759006 28118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:57:38.761240 28114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:38.765718 28118 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [term 2 LEADER]: Becoming Leader. State: Replica: 73425aedcb934f0c8b080a5c681a93fa, State: Running, Role: LEADER
I20260925 02:57:38.766335 28118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73425aedcb934f0c8b080a5c681a93fa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40393 } }
I20260925 02:57:38.769196 28119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "73425aedcb934f0c8b080a5c681a93fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73425aedcb934f0c8b080a5c681a93fa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40393 } } }
I20260925 02:57:38.770488 28119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:38.769431 28120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73425aedcb934f0c8b080a5c681a93fa. Latest consensus state: current_term: 2 leader_uuid: "73425aedcb934f0c8b080a5c681a93fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73425aedcb934f0c8b080a5c681a93fa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40393 } } }
I20260925 02:57:38.770744 28120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:38.776031 28129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:38.782850 28129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:38.784296 28129 catalog_manager.cc:1259] Loaded cluster ID: 5209ab80f45d446986f6e765d004a227
I20260925 02:57:38.784448 28129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:38.788499 28129 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:38.790175 28129 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 73425aedcb934f0c8b080a5c681a93fa: Loaded TSK: 0
I20260925 02:57:38.791790 28129 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:57:39.076292 28116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:39.076561 28116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:39.076705 28116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:39.083986 28116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:39.084086 28116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:39.084250 28116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 02:57:39.093957 28116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.193.keytab
--principal=oryx/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:40783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=40987
--tserver_master_addrs=127.18.196.254:40393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:39.095367 28116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:39.096700 28116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:39.110930 28142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:39.110930 28143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:39.112337 28145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:39.112556 28116 server_base.cc:1034] running on GCE node
I20260925 02:57:39.113200 28116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:39.114252 28116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:39.115463 28116 hybrid_clock.cc:648] HybridClock initialized: now 1790305059115410 us; error 47 us; skew 500 ppm
I20260925 02:57:39.115778 28116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:39 dist-test-slave-8zvk krb5kdc[27524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305059, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:39.118813 28116 init.cc:377] Logged in from keytab as oryx/127.18.196.193@KRBTEST.COM (short username oryx)
I20260925 02:57:39.120357 28116 webserver.cc:460] Webserver started at http://127.18.196.193:40987/ using document root <none> and password file <none>
I20260925 02:57:39.120999 28116 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:39.121140 28116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:39.126672 28116 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:57:39.129305 28152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:39.130618 28116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:57:39.130771 28116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "2a8ebe349b0a4756a17dca196fa89659"
format_stamp: "Formatted at 2026-09-25 02:57:36 on dist-test-slave-8zvk"
server_key: "4481acc42473689737a6d2c295553a3c"
server_key_iv: "0dabd3fd978acd627959547c8c79af31"
server_key_version: "encryptionkey@0"
I20260925 02:57:39.131724 28116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:39.146597 28116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:39.151029 28116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:39.151294 28116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:39.152330 28116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:39.153685 28116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:39.153779 28116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:39.153935 28116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:39.154039 28116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:39.188544 28116 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:40783
I20260925 02:57:39.188627 28264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:40783 every 8 connection(s)
I20260925 02:57:39.189949 28116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 02:57:39.195468 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 28116
I20260925 02:57:39.196194 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:33749
--local_ip_for_outbound_sockets=127.18.196.194
--tserver_master_addrs=127.18.196.254:40393
--webserver_port=41389
--webserver_interface=127.18.196.194
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.196.194
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:57:39 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:39 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:57:39.211570 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:39.192986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:45603 (local address 127.18.196.254:40393)
0925 02:57:39.193830 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:39.193847 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:39.194071 (+ 224us) server_negotiation.cc:407] Connection header received
0925 02:57:39.196912 (+ 2841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:39.196932 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:39.197370 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:39.197778 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:39.199302 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:39.201247 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:39.202982 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:39.203711 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:39.203841 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:39.210649 (+ 6808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.193:45603: BlockingRecv error: recv got EOF from 127.18.196.193:45603 (error 108)
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":342,"threads_started":1}
I20260925 02:57:39.211637 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:39.193026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.193:45603)
0925 02:57:39.193891 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:39.193947 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:39.196175 (+ 2228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:39.197907 (+ 1732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:39.197915 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:39.198436 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:39.199093 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:39.199114 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:39.201422 (+ 2308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:39.201426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:39.202757 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:39.202770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:39.203975 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:39.203986 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:39.204341 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:39.205096 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:39.205133 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:39.210276 (+ 5143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":354,"threads_started":1}
W20260925 02:57:39.212337 28265 heartbeater.cc:644] Failed to heartbeat to 127.18.196.254:40393 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.196.254:40393: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Sep 25 02:57:39 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:39 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:57:39.225039 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:39.213128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:35589 (local address 127.18.196.254:40393)
0925 02:57:39.213392 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:39.213425 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:39.213477 (+ 52us) server_negotiation.cc:407] Connection header received
0925 02:57:39.213561 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:39.213569 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:39.213655 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:39.213758 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:39.214800 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:39.216065 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:39.218824 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:39.219456 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:39.219591 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:39.224873 (+ 5282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.193:35589: BlockingRecv error: recv got EOF from 127.18.196.193:35589 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
I20260925 02:57:39.225138 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:39.212975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.193:35589)
0925 02:57:39.213268 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:39.213291 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:39.213402 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:39.213835 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:39.213843 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:39.214380 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:39.214607 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:39.214624 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:39.216277 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:39.216286 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:39.218619 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:39.218641 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:39.219632 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:39.219641 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:39.219909 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:39.220587 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:39.220619 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:39.224719 (+ 4100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139}
Sep 25 02:57:39 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:39 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:39.235545 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:39.225900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.193:47427)
0925 02:57:39.226250 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:39.226269 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:39.226365 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:39.226732 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:39.226737 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:39.227166 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:39.227360 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:39.227372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:39.228508 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:39.228514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:39.230148 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:39.230163 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:39.230825 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:39.230854 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:39.231125 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:39.231681 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:39.231705 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:39.235230 (+ 3525us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233}
W20260925 02:57:39.235836 28265 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:57:39.235890 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:39.226215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:47427 (local address 127.18.196.254:40393)
0925 02:57:39.226363 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:39.226367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:39.226382 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:57:39.226466 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:39.226469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:39.226564 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:39.226681 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:39.227493 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:39.228364 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:39.230286 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:39.230685 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:39.230774 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:39.235389 (+ 4615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.193:47427: BlockingRecv error: recv got EOF from 127.18.196.193:47427 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 02:57:39.525975 28269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:39.526288 28269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:39.526433 28269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:39.533550 28269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:39.533689 28269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:39.533859 28269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 02:57:39.543066 28269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.194.keytab
--principal=oryx/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:33749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=41389
--tserver_master_addrs=127.18.196.254:40393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:39.544251 28269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:39.545519 28269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:39.558214 28275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:39.558277 28274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:39.558193 28277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:39.560035 28269 server_base.cc:1034] running on GCE node
I20260925 02:57:39.560567 28269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:39.561501 28269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:39.562718 28269 hybrid_clock.cc:648] HybridClock initialized: now 1790305059562660 us; error 47 us; skew 500 ppm
I20260925 02:57:39.563038 28269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:39 dist-test-slave-8zvk krb5kdc[27524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305059, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:39.565797 28269 init.cc:377] Logged in from keytab as oryx/127.18.196.194@KRBTEST.COM (short username oryx)
I20260925 02:57:39.567263 28269 webserver.cc:460] Webserver started at http://127.18.196.194:41389/ using document root <none> and password file <none>
I20260925 02:57:39.567914 28269 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:39.568084 28269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:39.572890 28269 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:57:39.575536 28285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:39.576751 28269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:57:39.576923 28269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "e8134832c481409aa563523bd1e8190e"
format_stamp: "Formatted at 2026-09-25 02:57:37 on dist-test-slave-8zvk"
server_key: "af0130634424f485be7ffc7f1e24def6"
server_key_iv: "0deafcd3af5d2dc99a6a8ce20e0bb8af"
server_key_version: "encryptionkey@0"
I20260925 02:57:39.577847 28269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:39.601033 28269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:39.604655 28269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:39.604913 28269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:39.605836 28269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:39.607096 28269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:39.607218 28269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:39.607349 28269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:39.607438 28269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:39.642972 28269 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:33749
I20260925 02:57:39.643049 28397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:33749 every 8 connection(s)
I20260925 02:57:39.644341 28269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 02:57:39.653890 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 28269
I20260925 02:57:39.654503 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:43967
--local_ip_for_outbound_sockets=127.18.196.195
--tserver_master_addrs=127.18.196.254:40393
--webserver_port=34133
--webserver_interface=127.18.196.195
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.196.195
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:57:39 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:39 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:57:39.661213 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:39.646733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:33849 (local address 127.18.196.254:40393)
0925 02:57:39.646937 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:39.646941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:39.648038 (+ 1097us) server_negotiation.cc:407] Connection header received
0925 02:57:39.649854 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:39.649861 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:39.649950 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:39.650060 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:39.651538 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:39.652253 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:39.654002 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:39.654471 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:39.654577 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:39.660985 (+ 6408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.194:33849: BlockingRecv error: recv got EOF from 127.18.196.194:33849 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:57:39.662163 28400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:39.647219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.194:33849)
0925 02:57:39.647880 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:39.647919 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:39.649582 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:39.650302 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:39.650310 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:39.650777 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:39.651373 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:39.651391 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:39.652406 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:39.652418 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:39.653799 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:39.653814 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:39.654707 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:39.654717 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:39.655086 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:39.655905 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:39.655940 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:39.660824 (+ 4884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":274,"threads_started":1}
W20260925 02:57:39.662889 28398 heartbeater.cc:644] Failed to heartbeat to 127.18.196.254:40393 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.196.254:40393: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Sep 25 02:57:39 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:39 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:39.671832 28400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:39.663570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.194:43179)
0925 02:57:39.663743 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:39.663763 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:39.663860 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:39.664195 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:39.664198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:39.664485 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:39.664618 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:39.664626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:39.666093 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:39.666096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:39.667066 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:39.667078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:39.668129 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:39.668133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:39.668283 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:39.668865 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:39.668881 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:39.671632 (+ 2751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 02:57:39.672197 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:39.663699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:43179 (local address 127.18.196.254:40393)
0925 02:57:39.663875 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:39.663881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:39.663903 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:57:39.663966 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:39.663970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:39.664050 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:39.664170 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:39.664753 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:39.665955 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:39.667208 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:39.667979 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:39.668141 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:39.671908 (+ 3767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.194:43179: BlockingRecv error: recv got EOF from 127.18.196.194:43179 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 02:57:39 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:39 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:39.679872 28400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:39.672624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.194:42653)
0925 02:57:39.672785 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:39.672801 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:39.672888 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:39.673204 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:39.673207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:39.673452 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:39.673591 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:39.673598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:39.674568 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:39.674573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:39.675529 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:39.675541 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:39.676210 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:39.676216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:39.676391 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:39.676955 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:39.676971 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:39.679599 (+ 2628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 02:57:39.679992 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:39.672730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:42653 (local address 127.18.196.254:40393)
0925 02:57:39.672917 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:39.672925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:39.672943 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:57:39.672996 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:39.673003 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:39.673074 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:39.673180 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:39.673726 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:39.674427 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:39.675683 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:39.676066 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:39.676190 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:39.679777 (+ 3587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.194:42653: BlockingRecv error: recv got EOF from 127.18.196.194:42653 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 02:57:39.680281 28398 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:57:39.972281 28401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:39.972568 28401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:39.972745 28401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:39.979676 28401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:39.979813 28401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:39.979974 28401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 02:57:39.989359 28401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/krb5kdc/oryx_127.18.196.195.keytab
--principal=oryx/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:43967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=34133
--tserver_master_addrs=127.18.196.254:40393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:39.990556 28401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:39.991802 28401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:40.004360 28406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:40.004405 28409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:40.004451 28407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:40.005097 28401 server_base.cc:1034] running on GCE node
I20260925 02:57:40.005635 28401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:40.006459 28401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:40.007647 28401 hybrid_clock.cc:648] HybridClock initialized: now 1790305060007587 us; error 51 us; skew 500 ppm
I20260925 02:57:40.007946 28401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:57:40 dist-test-slave-8zvk krb5kdc[27524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305060, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:40.010974 28401 init.cc:377] Logged in from keytab as oryx/127.18.196.195@KRBTEST.COM (short username oryx)
I20260925 02:57:40.012321 28401 webserver.cc:460] Webserver started at http://127.18.196.195:34133/ using document root <none> and password file <none>
I20260925 02:57:40.012908 28401 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:40.013015 28401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:40.017959 28401 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:57:40.020399 28416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:40.021644 28401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:57:40.021802 28401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "008e3dcb93d344a0b4165607cb6ab7db"
format_stamp: "Formatted at 2026-09-25 02:57:37 on dist-test-slave-8zvk"
server_key: "f3f0d00cf663f197d3381cc9d67a5300"
server_key_iv: "7537fd188ced8908f5a8e55aa33db527"
server_key_version: "encryptionkey@0"
I20260925 02:57:40.022730 28401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:40.040082 28401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:40.044011 28401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:40.044287 28401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:40.045189 28401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:40.046432 28401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:40.046550 28401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:40.046671 28401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:40.046754 28401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:40.078902 28401 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:43967
I20260925 02:57:40.078982 28528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:43967 every 8 connection(s)
I20260925 02:57:40.080271 28401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 02:57:40.090124 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 28401
Sep 25 02:57:40 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:40 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:57:40.095722 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:40.082900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:54827 (local address 127.18.196.254:40393)
0925 02:57:40.083066 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:40.083070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:40.084171 (+ 1101us) server_negotiation.cc:407] Connection header received
0925 02:57:40.085885 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:40.085890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:40.085958 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:40.086045 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:40.087505 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:40.088224 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:40.089661 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:40.090323 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:40.090456 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:40.095445 (+ 4989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.195:54827: BlockingRecv error: recv got EOF from 127.18.196.195:54827 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:57:40.096192 28531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:40.083349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.195:54827)
0925 02:57:40.084006 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:40.084048 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:40.085642 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:40.086253 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:40.086260 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:40.086735 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:40.087356 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:40.087374 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:40.088349 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:40.088353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:40.089535 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:40.089548 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:40.090520 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:40.090525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:40.090732 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:40.091530 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:40.091569 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:40.095199 (+ 3630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":274,"threads_started":1}
W20260925 02:57:40.096817 28529 heartbeater.cc:644] Failed to heartbeat to 127.18.196.254:40393 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.196.254:40393: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Sep 25 02:57:40 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:40 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:40.105367 28531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:40.097401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.195:38581)
0925 02:57:40.097599 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:40.097619 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:40.097721 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:40.098076 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:40.098082 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:40.098505 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:40.098696 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:40.098711 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:40.099669 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:40.099675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:40.100719 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:40.100731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:40.101537 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:40.101544 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:40.101742 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:40.102165 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:40.102187 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:40.105186 (+ 2999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 02:57:40.105655 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:40.097530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:38581 (local address 127.18.196.254:40393)
0925 02:57:40.097665 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:40.097668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:40.097683 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:57:40.097849 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:40.097856 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:40.097929 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:40.098016 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:40.098875 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:40.099531 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:40.100850 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:40.101372 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:40.101485 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:40.105394 (+ 3909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.195:38581: BlockingRecv error: recv got EOF from 127.18.196.195:38581 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:57:40.107290 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:40.091220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45158 (local address 127.18.196.254:40393)
0925 02:57:40.092039 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:40.092045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:40.092073 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:57:40.092125 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:40.092128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:40.092178 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:40.092290 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:40.093863 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:40.094660 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:40.096297 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:40.096924 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:40.097049 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:40.100170 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:40.100178 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:40.100193 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:40.100227 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:40.103830 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:40.104494 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:40.104501 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:40.104508 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:40.104736 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:40.105117 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:40.105122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:40.105126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:40.105738 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:40.106150 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:40.106607 (+ 457us) server_negotiation.cc:299] Negotiation successful
0925 02:57:40.107002 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":567,"threads_started":1}
Sep 25 02:57:40 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:40 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:40.114398 28531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:40.106101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.195:38305)
0925 02:57:40.106946 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:40.106961 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:40.107077 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:40.107400 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:40.107405 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:40.107706 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:40.107837 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:40.107845 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:40.109152 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:40.109156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:40.110302 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:40.110314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:40.111141 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:40.111145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:40.111288 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:40.111815 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:40.111830 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:40.114125 (+ 2295us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":701}
W20260925 02:57:40.114557 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:40.106371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:38305 (local address 127.18.196.254:40393)
0925 02:57:40.106635 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:40.106641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:40.107028 (+ 387us) server_negotiation.cc:407] Connection header received
0925 02:57:40.107188 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:40.107192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:40.107275 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:40.107376 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:40.107971 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:40.109015 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:40.110428 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:40.111022 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:40.111118 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:40.114307 (+ 3189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.195:38305: BlockingRecv error: recv got EOF from 127.18.196.195:38305 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
W20260925 02:57:40.114751 28529 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 02:57:40 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:40 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:40.244865 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:40.236412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.193:50215)
0925 02:57:40.236939 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:40.236956 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:40.237065 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:40.237348 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:40.237351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:40.237800 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:40.237992 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:40.238000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:40.239034 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:40.239039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:40.240498 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:40.240513 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:40.241296 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:40.241300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:40.241445 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:40.242059 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:40.242076 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:40.244653 (+ 2577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":291,"threads_started":1}
W20260925 02:57:40.245216 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:40.236483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:50215 (local address 127.18.196.254:40393)
0925 02:57:40.236654 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:40.236658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:40.237025 (+ 367us) server_negotiation.cc:407] Connection header received
0925 02:57:40.237159 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:40.237163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:40.237223 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:40.237304 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:40.238110 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:40.238871 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:40.240652 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:40.241143 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:40.241237 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:40.244856 (+ 3619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.193:50215: BlockingRecv error: recv got EOF from 127.18.196.193:50215 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 02:57:40 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:40 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:40.689348 28535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:40.680959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.194:42937)
0925 02:57:40.681547 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:40.681568 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:40.681707 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:40.682031 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:40.682034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:40.682461 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:40.682725 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:40.682736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:40.683910 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:40.683914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:40.684857 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:40.684867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:40.685622 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:40.685627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:40.685745 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:40.686298 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:40.686315 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:40.689093 (+ 2778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":295,"threads_started":1}
W20260925 02:57:40.689579 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:40.681083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:42937 (local address 127.18.196.254:40393)
0925 02:57:40.681271 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:40.681275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:40.681639 (+ 364us) server_negotiation.cc:407] Connection header received
0925 02:57:40.681825 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:40.681829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:40.681890 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:40.681972 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:40.682974 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:40.683746 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:40.685003 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:40.685443 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:40.685537 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:40.689315 (+ 3778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.194:42937: BlockingRecv error: recv got EOF from 127.18.196.194:42937 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 02:57:41 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:41 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:41.124025 28536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:41.115359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.195:57307)
0925 02:57:41.115926 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:41.115956 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:41.116065 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:41.116369 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:41.116373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:41.116793 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:41.117024 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:41.117038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:41.118508 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:41.118515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:41.119578 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:41.119589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:41.120278 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:41.120282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:41.120395 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:41.120923 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:41.120947 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:41.123704 (+ 2757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":331,"threads_started":1}
W20260925 02:57:41.124102 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:41.115476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:57307 (local address 127.18.196.254:40393)
0925 02:57:41.115661 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:41.115665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:41.116034 (+ 369us) server_negotiation.cc:407] Connection header received
0925 02:57:41.116162 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:41.116166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:41.116230 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:41.116313 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:41.117195 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:41.118297 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:41.119724 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:41.120116 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:41.120206 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:41.123882 (+ 3676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.195:57307: BlockingRecv error: recv got EOF from 127.18.196.195:57307 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 02:57:41 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:41 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:41.254340 28537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:41.245775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.193:36613)
0925 02:57:41.246268 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:41.246283 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:41.246447 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:41.246894 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:41.246898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:41.247429 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:41.247690 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:41.247703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:41.248739 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:41.248743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:41.249661 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:41.249672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:41.250350 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:41.250355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:41.250490 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:41.251095 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:41.251113 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:41.253983 (+ 2870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":276,"threads_started":1}
W20260925 02:57:41.254443 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:41.245971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:36613 (local address 127.18.196.254:40393)
0925 02:57:41.246163 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:41.246170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:41.246387 (+ 217us) server_negotiation.cc:407] Connection header received
0925 02:57:41.246584 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:41.246589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:41.246700 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:41.246803 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:41.247848 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:41.248622 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:41.249781 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:41.250189 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:41.250287 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:41.254172 (+ 3885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.193:36613: BlockingRecv error: recv got EOF from 127.18.196.193:36613 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 02:57:41 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:41 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:41.698463 28538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:41.690321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.194:47583)
0925 02:57:41.691033 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:41.691058 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:41.691209 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:41.691553 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:41.691557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:41.691995 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:41.692155 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:41.692164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:41.693135 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:41.693142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:41.694098 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:41.694111 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:41.694803 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:41.694808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:41.695009 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:41.695505 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:41.695521 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:41.698233 (+ 2712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":411,"threads_started":1}
W20260925 02:57:41.698642 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:41.690407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:47583 (local address 127.18.196.254:40393)
0925 02:57:41.690686 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:41.690695 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:41.691146 (+ 451us) server_negotiation.cc:407] Connection header received
0925 02:57:41.691341 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:41.691346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:41.691422 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:41.691521 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:41.692296 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:41.693009 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:41.694239 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:41.694650 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:41.694775 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:41.698427 (+ 3652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.194:47583: BlockingRecv error: recv got EOF from 127.18.196.194:47583 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 25 02:57:42 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:42 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:42.133016 28539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:42.124938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.195:40451)
0925 02:57:42.125423 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:42.125440 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:42.125549 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:42.125862 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:42.125865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:42.126351 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:42.126531 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:42.126540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:42.127642 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:42.127646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:42.128531 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:42.128541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:42.129144 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:42.129148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:42.129252 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:42.129791 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:42.129807 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:42.132684 (+ 2877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":251,"threads_started":1}
W20260925 02:57:42.133191 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:42.125140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:40451 (local address 127.18.196.254:40393)
0925 02:57:42.125350 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:42.125358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:42.125512 (+ 154us) server_negotiation.cc:407] Connection header received
0925 02:57:42.125643 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:42.125646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:42.125708 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:42.125802 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:42.126655 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:42.127524 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:42.128643 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:42.129025 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:42.129109 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:42.132879 (+ 3770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.195:40451: BlockingRecv error: recv got EOF from 127.18.196.195:40451 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 25 02:57:42 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:42 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:42.263737 28540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:42.255349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.193:36067)
0925 02:57:42.255869 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:42.255885 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:42.255997 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:42.256288 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:42.256292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:42.256636 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:42.256834 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:42.256844 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:42.257940 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:42.257944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:42.258918 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:42.258931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:42.259622 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:42.259627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:42.259761 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:42.260328 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:42.260346 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:42.263398 (+ 3052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":301,"threads_started":1}
W20260925 02:57:42.264104 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:42.255440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:36067 (local address 127.18.196.254:40393)
0925 02:57:42.255626 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:42.255634 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:42.255953 (+ 319us) server_negotiation.cc:407] Connection header received
0925 02:57:42.256095 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:42.256099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:42.256157 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:42.256237 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:42.256953 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:42.257748 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:42.259046 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:42.259464 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:42.259567 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:42.263642 (+ 4075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.193:36067: BlockingRecv error: recv got EOF from 127.18.196.193:36067 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 02:57:42 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:42 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:42.716636 28541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:42.699541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.194:50317)
0925 02:57:42.700759 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:42.700786 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:42.700950 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:42.701351 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:42.701357 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:42.701852 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:42.702037 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:42.702046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:42.703534 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:42.703538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:42.704466 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:42.704479 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:42.705387 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:42.705391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:42.705529 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:42.713164 (+ 7635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:42.713189 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:42.716270 (+ 3081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1101,"thread_start_us":792,"threads_started":1}
W20260925 02:57:42.716760 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:42.699659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:50317 (local address 127.18.196.254:40393)
0925 02:57:42.699909 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:42.699924 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:42.700883 (+ 959us) server_negotiation.cc:407] Connection header received
0925 02:57:42.701106 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:42.701111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:42.701188 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:42.701286 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:42.702171 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:42.703397 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:42.704610 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:42.705250 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:42.705377 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:42.716515 (+ 11138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.194:50317: BlockingRecv error: recv got EOF from 127.18.196.194:50317 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 25 02:57:43 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:43 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:43.142357 28542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:43.134054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.195:44131)
0925 02:57:43.134501 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:43.134526 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:43.134669 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:43.135010 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:43.135013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:43.135360 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:43.135598 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:43.135608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:43.136841 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:43.136845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:43.137915 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:43.137928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:43.138577 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:43.138581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:43.138705 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:43.139360 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:43.139377 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:43.142111 (+ 2734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":247,"threads_started":1}
W20260925 02:57:43.142504 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:43.134125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:44131 (local address 127.18.196.254:40393)
0925 02:57:43.134356 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:43.134365 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:43.134609 (+ 244us) server_negotiation.cc:407] Connection header received
0925 02:57:43.134809 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:43.134813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:43.134894 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:43.134980 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:43.135720 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:43.136696 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:43.138045 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:43.138446 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:43.138543 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:43.142299 (+ 3756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.195:44131: BlockingRecv error: recv got EOF from 127.18.196.195:44131 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 02:57:43 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:43 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:43.273073 28543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:43.264782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.193:59503)
0925 02:57:43.265242 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:43.265258 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:43.265370 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:43.265664 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:43.265668 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:43.266021 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:43.266230 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:43.266244 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:43.267600 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:43.267608 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:43.268695 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:43.268707 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:43.269370 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:43.269374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:43.269485 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:43.270023 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:43.270040 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:43.272736 (+ 2696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":267,"threads_started":1}
W20260925 02:57:43.273173 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:43.264812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:59503 (local address 127.18.196.254:40393)
0925 02:57:43.265082 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:43.265089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:43.265336 (+ 247us) server_negotiation.cc:407] Connection header received
0925 02:57:43.265473 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:43.265477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:43.265539 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:43.265622 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:43.266412 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:43.267436 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:43.268819 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:43.269250 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:43.269344 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:43.272924 (+ 3580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.193:59503: BlockingRecv error: recv got EOF from 127.18.196.193:59503 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 25 02:57:43 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:43 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:43.725368 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:43.717763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.194:45495)
0925 02:57:43.718301 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:43.718328 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:43.718434 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:43.718707 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:43.718710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:43.719080 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:43.719239 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:43.719247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:43.720131 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:43.720134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:43.721054 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:43.721065 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:43.721714 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:43.721718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:43.721833 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:43.722375 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:43.722392 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:43.725078 (+ 2686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":304,"threads_started":1}
W20260925 02:57:43.725514 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:43.717810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:45495 (local address 127.18.196.254:40393)
0925 02:57:43.717994 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:43.717999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:43.718397 (+ 398us) server_negotiation.cc:407] Connection header received
0925 02:57:43.718531 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:43.718534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:43.718590 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:43.718664 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:43.719351 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:43.720023 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:43.721179 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:43.721595 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:43.721684 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:43.725279 (+ 3595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.194:45495: BlockingRecv error: recv got EOF from 127.18.196.194:45495 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 02:57:44 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:44 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:44.151757 28545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:44.143325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.195:38953)
0925 02:57:44.143848 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:44.143865 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:44.143981 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:44.144272 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:44.144276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:44.144646 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:44.144799 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:44.144808 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:44.145907 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:44.145912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:44.146947 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:44.146961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:44.147628 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:44.147631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:44.147750 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:44.148401 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:44.148420 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:44.151431 (+ 3011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":295,"threads_started":1}
W20260925 02:57:44.151916 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:44.143464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:38953 (local address 127.18.196.254:40393)
0925 02:57:44.143654 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:44.143659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:44.143940 (+ 281us) server_negotiation.cc:407] Connection header received
0925 02:57:44.144077 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:44.144081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:44.144142 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:44.144227 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:44.144935 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:44.145744 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:44.147099 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:44.147500 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:44.147592 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:44.151620 (+ 4028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.195:38953: BlockingRecv error: recv got EOF from 127.18.196.195:38953 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 02:57:44 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:44 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:44.282161 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:44.274058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.193:37659)
0925 02:57:44.274610 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:44.274626 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:44.274734 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:44.275038 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:44.275041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:44.275465 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:44.275620 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:44.275628 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:44.276676 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:44.276680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:44.277581 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:44.277592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:44.278254 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:44.278258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:44.278383 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:44.279046 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:44.279063 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:44.281852 (+ 2789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":281,"threads_started":1}
W20260925 02:57:44.282269 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:44.274151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:37659 (local address 127.18.196.254:40393)
0925 02:57:44.274343 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:44.274347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:44.274698 (+ 351us) server_negotiation.cc:407] Connection header received
0925 02:57:44.274827 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:44.274850 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:44.274909 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:44.274994 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:44.275773 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:44.276542 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:44.277713 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:44.278120 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:44.278212 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:44.282058 (+ 3846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.193:37659: BlockingRecv error: recv got EOF from 127.18.196.193:37659 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 02:57:44 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:44 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:44.734732 28547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:44.726535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.194:40197)
0925 02:57:44.727157 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:44.727182 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:44.727321 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:44.727708 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:44.727712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:44.728228 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:44.728381 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:44.728390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:44.729347 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:44.729350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:44.730238 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:44.730249 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:44.730933 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:44.730937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:44.731061 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:44.731622 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:44.731639 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:44.734487 (+ 2848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":240,"threads_started":1}
W20260925 02:57:44.735016 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:44.726701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:40197 (local address 127.18.196.254:40393)
0925 02:57:44.726944 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:44.726950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:44.727269 (+ 319us) server_negotiation.cc:407] Connection header received
0925 02:57:44.727453 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:44.727459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:44.727553 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:44.727644 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:44.728511 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:44.729232 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:44.730353 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:44.730776 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:44.730930 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:44.734717 (+ 3787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.194:40197: BlockingRecv error: recv got EOF from 127.18.196.194:40197 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 25 02:57:45 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:45 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:45.161265 28548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:45.152780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.195:53555)
0925 02:57:45.153398 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:45.153419 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:45.153557 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:45.153982 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:45.153987 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:45.154583 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:45.154805 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:45.154819 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:45.155858 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:45.155862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:45.156861 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:45.156872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:45.157589 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:45.157593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:45.157726 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:45.158394 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:45.158412 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:45.161021 (+ 2609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":277,"threads_started":1}
W20260925 02:57:45.161455 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:45.152985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:53555 (local address 127.18.196.254:40393)
0925 02:57:45.153207 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:45.153213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:45.153536 (+ 323us) server_negotiation.cc:407] Connection header received
0925 02:57:45.153676 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:45.153682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:45.153777 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:45.153901 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:45.155005 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:45.155727 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:45.156982 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:45.157459 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:45.157556 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:45.161213 (+ 3657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.195:53555: BlockingRecv error: recv got EOF from 127.18.196.195:53555 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 25 02:57:45 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:45 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:45.291257 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:45.283188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.193:51159)
0925 02:57:45.283784 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:45.283805 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:45.283972 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:45.284373 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:45.284378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:45.284957 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:45.285177 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:45.285192 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:45.286180 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:45.286184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:45.287075 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:45.287085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:45.287743 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:45.287747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:45.287859 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:45.288419 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:45.288436 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:45.290968 (+ 2532us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":316,"threads_started":1}
W20260925 02:57:45.291388 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:45.283273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:51159 (local address 127.18.196.254:40393)
0925 02:57:45.283524 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:45.283532 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:45.283903 (+ 371us) server_negotiation.cc:407] Connection header received
0925 02:57:45.284100 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:45.284106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:45.284210 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:45.284311 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:45.285321 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:45.286065 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:45.287216 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:45.287612 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:45.287710 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:45.291176 (+ 3466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.193:51159: BlockingRecv error: recv got EOF from 127.18.196.193:51159 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 25 02:57:45 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:45 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:45.744035 28550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:45.735746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.194:43193)
0925 02:57:45.736239 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:45.736256 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:45.736404 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:45.736735 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:45.736738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:45.737094 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:45.737253 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:45.737261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:45.738171 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:45.738174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:45.739220 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:45.739232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:45.739891 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:45.739895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:45.740009 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:45.740607 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:45.740624 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:45.743728 (+ 3104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":284,"threads_started":1}
W20260925 02:57:45.744207 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:45.735785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:43193 (local address 127.18.196.254:40393)
0925 02:57:45.736009 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:45.736018 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:45.736366 (+ 348us) server_negotiation.cc:407] Connection header received
0925 02:57:45.736495 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:45.736498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:45.736603 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:45.736675 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:45.737369 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:45.738056 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:45.739339 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:45.739764 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:45.739852 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:45.743953 (+ 4101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.194:43193: BlockingRecv error: recv got EOF from 127.18.196.194:43193 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 02:57:46 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:46 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:46.170357 28551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:46.162230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.195:37041)
0925 02:57:46.162782 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:46.162801 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:46.162971 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:46.163281 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:46.163286 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:46.163694 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:46.163847 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:46.163856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:46.164960 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:46.164967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:46.165838 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:46.165849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:46.166612 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:46.166616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:46.166735 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:46.167267 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:46.167300 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:46.169976 (+ 2676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":299,"threads_started":1}
W20260925 02:57:46.170579 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:46.162297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:37041 (local address 127.18.196.254:40393)
0925 02:57:46.162530 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:46.162538 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:46.162900 (+ 362us) server_negotiation.cc:407] Connection header received
0925 02:57:46.163088 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:46.163093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:46.163157 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:46.163249 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:46.164001 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:46.164792 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:46.166008 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:46.166431 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:46.166569 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:46.170277 (+ 3708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.195:37041: BlockingRecv error: recv got EOF from 127.18.196.195:37041 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 02:57:46 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:46 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:46.300848 28552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:46.292182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.193:50397)
0925 02:57:46.292799 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:46.292818 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:46.292966 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:46.293322 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:46.293327 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:46.293856 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:46.294036 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:46.294045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:46.295082 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:46.295087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:46.296019 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:46.296030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:46.296784 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:46.296790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:46.296966 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:46.297505 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:46.297523 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:46.300613 (+ 3090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":264,"threads_started":1}
W20260925 02:57:46.301026 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:46.292265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:50397 (local address 127.18.196.254:40393)
0925 02:57:46.292536 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:46.292543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:46.292902 (+ 359us) server_negotiation.cc:407] Connection header received
0925 02:57:46.293095 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:46.293100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:46.293180 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:46.293294 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:46.294177 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:46.294947 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:46.296161 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:46.296613 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:46.296771 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:46.300817 (+ 4046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.193:50397: BlockingRecv error: recv got EOF from 127.18.196.193:50397 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 02:57:46 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:46 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:46.753690 28553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:46.744986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.194:42589)
0925 02:57:46.745826 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:46.745851 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:46.746018 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:46.746450 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:46.746455 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:46.747095 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:46.747285 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:46.747295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:46.748342 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:46.748346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:46.749266 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:46.749277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:46.750005 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:46.750009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:46.750122 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:46.750717 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:46.750736 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:46.753431 (+ 2695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":482,"threads_started":1}
W20260925 02:57:46.754014 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:46.745255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:42589 (local address 127.18.196.254:40393)
0925 02:57:46.745504 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:46.745512 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:46.745959 (+ 447us) server_negotiation.cc:407] Connection header received
0925 02:57:46.746159 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:46.746164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:46.746274 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:46.746392 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:46.747415 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:46.748198 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:46.749422 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:46.749857 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:46.749968 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:46.753648 (+ 3680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.194:42589: BlockingRecv error: recv got EOF from 127.18.196.194:42589 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Sep 25 02:57:47 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:47 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:47.180335 28554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:47.171592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.195:50631)
0925 02:57:47.172178 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:47.172195 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:47.172309 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:47.172606 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:47.172609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:47.173020 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:47.173186 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:47.173194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:47.174412 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:47.174421 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:47.175589 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:47.175607 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:47.176358 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:47.176364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:47.176538 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:47.177152 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:47.177170 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:47.179983 (+ 2813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":293,"threads_started":1}
W20260925 02:57:47.180423 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:47.171615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:50631 (local address 127.18.196.254:40393)
0925 02:57:47.171789 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:47.171794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:47.172272 (+ 478us) server_negotiation.cc:407] Connection header received
0925 02:57:47.172407 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:47.172410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:47.172472 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:47.172555 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:47.173358 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:47.174248 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:47.175752 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:47.176195 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:47.176331 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:47.180195 (+ 3864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.195:50631: BlockingRecv error: recv got EOF from 127.18.196.195:50631 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 02:57:47 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:47 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:47.311056 28555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:47.302111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.193:38219)
0925 02:57:47.302738 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:47.302756 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:47.302952 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:47.303282 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:47.303285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:47.303727 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:47.303894 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:47.303904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:47.304955 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:47.304961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:47.306186 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:47.306203 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:47.307139 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:47.307146 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:47.307328 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:47.307906 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:47.307924 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:47.310645 (+ 2721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":355,"threads_started":1}
W20260925 02:57:47.311173 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:47.302502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:38219 (local address 127.18.196.254:40393)
0925 02:57:47.302702 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:47.302707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:47.302815 (+ 108us) server_negotiation.cc:407] Connection header received
0925 02:57:47.303082 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:47.303086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:47.303147 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:47.303225 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:47.304040 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:47.304785 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:47.306318 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:47.306929 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:47.307068 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:47.310878 (+ 3810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.193:38219: BlockingRecv error: recv got EOF from 127.18.196.193:38219 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 02:57:47 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:47 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:47.771215 28556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:47.754941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.194:34459)
0925 02:57:47.755505 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:47.755524 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:47.755685 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:47.756030 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:47.756038 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:47.756519 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:47.756668 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:47.756678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:47.758397 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:47.758401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:47.759330 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:47.759341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:47.760018 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:47.760023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:47.760133 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:47.766285 (+ 6152us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:47.766308 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:47.770882 (+ 4574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":278,"threads_started":1}
W20260925 02:57:47.771360 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:47.755029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:34459 (local address 127.18.196.254:40393)
0925 02:57:47.755236 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:47.755243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:47.755614 (+ 371us) server_negotiation.cc:407] Connection header received
0925 02:57:47.755821 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:47.755826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:47.755886 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:47.755989 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:47.756855 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:47.758270 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:47.759443 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:47.759861 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:47.759952 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:47.771114 (+ 11162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.194:34459: BlockingRecv error: recv got EOF from 127.18.196.194:34459 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 02:57:48 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:48 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:48.190194 28557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:48.181385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.195:37553)
0925 02:57:48.181943 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:48.181959 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:48.182127 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:48.182501 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:48.182506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:48.182985 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:48.183146 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:48.183156 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:48.184244 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:48.184250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:48.185155 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:48.185167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:48.185918 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:48.185924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:48.186091 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:48.186788 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:48.186805 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:48.189862 (+ 3057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":271,"threads_started":1}
W20260925 02:57:48.190358 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:48.181489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:37553 (local address 127.18.196.254:40393)
0925 02:57:48.181727 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:48.181734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:48.182056 (+ 322us) server_negotiation.cc:407] Connection header received
0925 02:57:48.182287 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:48.182292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:48.182351 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:48.182464 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:48.183330 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:48.184065 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:48.185305 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:48.185723 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:48.185861 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:48.190105 (+ 4244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.195:37553: BlockingRecv error: recv got EOF from 127.18.196.195:37553 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 02:57:48 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:48 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:48.321143 28558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:48.312180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.193:35003)
0925 02:57:48.312725 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:48.312740 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:48.312901 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:48.313270 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:48.313275 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:48.313706 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:48.313868 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:48.313878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:48.314952 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:48.314961 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:48.316165 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:48.316181 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:48.316916 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:48.316922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:48.317103 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:48.317632 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:48.317649 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:48.320796 (+ 3147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":284,"threads_started":1}
W20260925 02:57:48.321226 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:48.312277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:35003 (local address 127.18.196.254:40393)
0925 02:57:48.312521 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:48.312527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:48.312834 (+ 307us) server_negotiation.cc:407] Connection header received
0925 02:57:48.313031 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:48.313037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:48.313127 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:48.313237 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:48.314024 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:48.314779 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:48.316347 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:48.316758 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:48.316898 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:48.320992 (+ 4094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.193:35003: BlockingRecv error: recv got EOF from 127.18.196.193:35003 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 02:57:48 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:48 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:48.780648 28559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:48.772315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.194:60699)
0925 02:57:48.772837 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:48.772861 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:48.773017 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:48.773356 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:48.773361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:48.773789 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:48.773950 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:48.773960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:48.775032 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:48.775040 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:48.775976 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:48.775988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:48.776731 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:48.776737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:48.776930 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:48.777518 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:48.777535 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:48.780417 (+ 2882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":279,"threads_started":1}
W20260925 02:57:48.780846 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:48.772381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:60699 (local address 127.18.196.254:40393)
0925 02:57:48.772608 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:48.772616 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:48.772951 (+ 335us) server_negotiation.cc:407] Connection header received
0925 02:57:48.773150 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:48.773162 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:48.773221 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:48.773329 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:48.774109 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:48.774896 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:48.776146 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:48.776566 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:48.776709 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:48.780588 (+ 3879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.194:60699: BlockingRecv error: recv got EOF from 127.18.196.194:60699 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 02:57:49 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:49 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:49.199621 28560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:49.191169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.195:35551)
0925 02:57:49.191889 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:49.191904 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:49.192012 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:49.192298 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:49.192301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:49.192661 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:49.192821 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:49.192829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:49.193818 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:49.193822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:49.194934 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:49.194949 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:49.195739 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:49.195744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:49.195860 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:49.196390 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:49.196407 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:49.199269 (+ 2862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":481,"threads_started":1}
W20260925 02:57:49.199671 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:49.191234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:35551 (local address 127.18.196.254:40393)
0925 02:57:49.191463 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:49.191472 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:49.191990 (+ 518us) server_negotiation.cc:407] Connection header received
0925 02:57:49.192110 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:49.192114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:49.192173 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:49.192250 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:49.192963 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:49.193666 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:49.195071 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:49.195577 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:49.195690 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:49.199446 (+ 3756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.195:35551: BlockingRecv error: recv got EOF from 127.18.196.195:35551 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 02:57:49 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:49 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:49.330654 28561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:49.322330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.193:34071)
0925 02:57:49.322862 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:49.322876 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:49.323030 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:49.323525 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:49.323529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:49.323871 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:49.324002 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:49.324009 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:49.325139 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:49.325161 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:49.326233 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:49.326245 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:49.327050 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:49.327055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:49.327168 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:49.327729 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:49.327746 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:49.330442 (+ 2696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":298,"threads_started":1}
W20260925 02:57:49.330890 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:49.322449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:34071 (local address 127.18.196.254:40393)
0925 02:57:49.322705 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:49.322711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:49.322964 (+ 253us) server_negotiation.cc:407] Connection header received
0925 02:57:49.323147 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:49.323152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:49.323232 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:49.323340 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:49.324162 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:49.324990 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:49.326373 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:49.326910 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:49.327026 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:49.330604 (+ 3578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.193:34071: BlockingRecv error: recv got EOF from 127.18.196.193:34071 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 02:57:49 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:49 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:49.789618 28562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:49.781725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.194:50419)
0925 02:57:49.782255 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:49.782276 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:49.782384 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:49.782687 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:49.782690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:49.783061 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:49.783193 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:49.783201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:49.784171 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:49.784175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:49.785074 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:49.785085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:49.785786 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:49.785790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:49.785902 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:49.786555 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:49.786573 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:49.789405 (+ 2832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":313,"threads_started":1}
W20260925 02:57:49.789839 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:49.781787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:50419 (local address 127.18.196.254:40393)
0925 02:57:49.782061 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:49.782065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:49.782347 (+ 282us) server_negotiation.cc:407] Connection header received
0925 02:57:49.782478 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:49.782482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:49.782542 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:49.782624 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:49.783312 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:49.784039 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:49.785210 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:49.785651 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:49.785743 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:49.789585 (+ 3842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.194:50419: BlockingRecv error: recv got EOF from 127.18.196.194:50419 (error 108)
Metrics: {"server-negotiator.queue_time_us":152}
Sep 25 02:57:50 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:50 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:50.208820 28563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:50.200636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.195:34961)
0925 02:57:50.201145 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:50.201161 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:50.201270 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:50.201610 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:50.201613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:50.201951 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:50.202090 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:50.202138 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:50.203560 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:50.203567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:50.204468 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:50.204479 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:50.205135 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:50.205139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:50.205240 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:50.205759 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:50.205773 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:50.208478 (+ 2705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":274,"threads_started":1}
W20260925 02:57:50.208935 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:50.200916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:34961 (local address 127.18.196.254:40393)
0925 02:57:50.201120 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:50.201127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:50.201233 (+ 106us) server_negotiation.cc:407] Connection header received
0925 02:57:50.201361 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:50.201364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:50.201455 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:50.201552 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:50.202307 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:50.203418 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:50.204622 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:50.205014 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:50.205099 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:50.208732 (+ 3633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.195:34961: BlockingRecv error: recv got EOF from 127.18.196.195:34961 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 02:57:50 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:50 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:50.339809 28564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:50.331639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.193:46701)
0925 02:57:50.332221 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:50.332267 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:50.332406 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:50.332792 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:50.332797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:50.333235 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:50.333403 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:50.333412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:50.334417 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:50.334424 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:50.335402 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:50.335414 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:50.336115 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:50.336120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:50.336303 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:50.336834 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:50.336850 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:50.339577 (+ 2727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":360,"threads_started":1}
W20260925 02:57:50.339967 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:50.331821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:46701 (local address 127.18.196.254:40393)
0925 02:57:50.332086 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:50.332092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:50.332344 (+ 252us) server_negotiation.cc:407] Connection header received
0925 02:57:50.332541 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:50.332546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:50.332625 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:50.332760 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:50.333555 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:50.334260 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:50.335571 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:50.335963 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:50.336106 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:50.339770 (+ 3664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.193:46701: BlockingRecv error: recv got EOF from 127.18.196.193:46701 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 02:57:50 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:50 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:50.798939 28565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:50.790626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.194:60373)
0925 02:57:50.791185 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:50.791206 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:50.791371 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:50.791738 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:50.791741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:50.792127 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:50.792301 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:50.792309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:50.793281 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:50.793285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:50.794180 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:50.794192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:50.794887 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:50.794894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:50.795033 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:50.795642 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:50.795665 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:50.798553 (+ 2888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":319,"threads_started":1}
W20260925 02:57:50.799028 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:50.790718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:60373 (local address 127.18.196.254:40393)
0925 02:57:50.790940 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:50.790948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:50.791297 (+ 349us) server_negotiation.cc:407] Connection header received
0925 02:57:50.791541 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:50.791545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:50.791606 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:50.791684 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:50.792421 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:50.793151 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:50.794303 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:50.794705 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:50.794803 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:50.798765 (+ 3962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.194:60373: BlockingRecv error: recv got EOF from 127.18.196.194:60373 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 02:57:51 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:51 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:51.219085 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:51.209872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.195:46261)
0925 02:57:51.210424 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:51.210440 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:51.210607 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:51.211020 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:51.211024 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:51.211529 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:51.211704 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:51.211714 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:51.212765 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:51.212771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:51.213747 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:51.213759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:51.214482 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:51.214488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:51.214667 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:51.215377 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:51.215395 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:51.218712 (+ 3317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":308,"threads_started":1}
W20260925 02:57:51.219213 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:51.210000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:46261 (local address 127.18.196.254:40393)
0925 02:57:51.210250 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:51.210256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:51.210537 (+ 281us) server_negotiation.cc:407] Connection header received
0925 02:57:51.210736 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:51.210744 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:51.210819 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:51.210984 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:51.211862 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:51.212623 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:51.213898 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:51.214311 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:51.214455 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:51.218972 (+ 4517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.195:46261: BlockingRecv error: recv got EOF from 127.18.196.195:46261 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 25 02:57:51 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:51 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:51.349371 28567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:51.340651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.193:48627)
0925 02:57:51.341171 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:51.341186 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:51.341342 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:51.341747 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:51.341751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:51.342202 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:51.342417 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:51.342430 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:51.343699 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:51.343703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:51.344799 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:51.344811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:51.345516 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:51.345520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:51.345641 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:51.346194 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:51.346210 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:51.349145 (+ 2935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":303,"threads_started":1}
W20260925 02:57:51.349613 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:51.340715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:48627 (local address 127.18.196.254:40393)
0925 02:57:51.340972 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:51.340980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:51.341273 (+ 293us) server_negotiation.cc:407] Connection header received
0925 02:57:51.341539 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:51.341543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:51.341605 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:51.341712 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:51.342610 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:51.343530 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:51.344921 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:51.345394 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:51.345488 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:51.349336 (+ 3848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.193:48627: BlockingRecv error: recv got EOF from 127.18.196.193:48627 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 02:57:51 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:51 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:51.808631 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:51.799977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.194:43611)
0925 02:57:51.800476 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:51.800501 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:51.800652 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:51.800995 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:51.800999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:51.801461 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:51.801619 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:51.801628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:51.802635 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:51.802643 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:51.803593 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:51.803606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:51.804309 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:51.804314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:51.804499 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:51.805169 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:51.805186 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:51.808306 (+ 3120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":291,"threads_started":1}
W20260925 02:57:51.808730 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:51.800030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:43611 (local address 127.18.196.254:40393)
0925 02:57:51.800272 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:51.800279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:51.800595 (+ 316us) server_negotiation.cc:407] Connection header received
0925 02:57:51.800788 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:51.800797 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:51.800853 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:51.800953 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:51.801771 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:51.802484 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:51.803741 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:51.804157 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:51.804297 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:51.808515 (+ 4218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.194:43611: BlockingRecv error: recv got EOF from 127.18.196.194:43611 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 02:57:52 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:52 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:52.229202 28569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:52.220684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.195:38543)
0925 02:57:52.221203 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:52.221220 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:52.221329 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:52.221665 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:52.221672 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:52.222150 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:52.222308 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:52.222317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:52.223631 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:52.223637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:52.224799 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:52.224812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:52.225445 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:52.225449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:52.225570 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:52.226122 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:52.226138 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:52.228931 (+ 2793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":290,"threads_started":1}
W20260925 02:57:52.229343 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:52.220764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:38543 (local address 127.18.196.254:40393)
0925 02:57:52.220935 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:52.220939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:52.221294 (+ 355us) server_negotiation.cc:407] Connection header received
0925 02:57:52.221439 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:52.221443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:52.221512 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:52.221595 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:52.222434 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:52.223473 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:52.224934 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:52.225326 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:52.225416 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:52.229131 (+ 3715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.195:38543: BlockingRecv error: recv got EOF from 127.18.196.195:38543 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 02:57:52 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:52 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:52.359015 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:52.350423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.193:57209)
0925 02:57:52.351000 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:52.351015 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:52.351122 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:52.351416 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:52.351419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:52.351807 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:52.351962 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:52.351970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:52.353187 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:52.353193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:52.354438 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:52.354451 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:52.355138 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:52.355142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:52.355261 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:52.355812 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:52.355827 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:52.358710 (+ 2883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":288,"threads_started":1}
W20260925 02:57:52.359202 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:52.350494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:57209 (local address 127.18.196.254:40393)
0925 02:57:52.350665 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:52.350670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:52.351088 (+ 418us) server_negotiation.cc:407] Connection header received
0925 02:57:52.351212 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:52.351216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:52.351289 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:52.351367 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:52.352108 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:52.353045 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:52.354565 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:52.354988 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:52.355080 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:52.358959 (+ 3879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.193:57209: BlockingRecv error: recv got EOF from 127.18.196.193:57209 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 02:57:52 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:52 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:52.826910 28571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:52.809585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.194:43807)
0925 02:57:52.810123 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:52.810140 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:52.810290 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:52.810644 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:52.810648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:52.811211 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:52.811370 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:52.811379 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:52.812386 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:52.812391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:52.813310 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:52.813321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:52.814039 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:52.814045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:52.814243 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:52.823697 (+ 9454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:52.823716 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:52.826539 (+ 2823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":295,"threads_started":1}
W20260925 02:57:52.826993 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:52.809785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:43807 (local address 127.18.196.254:40393)
0925 02:57:52.809995 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:52.810001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:52.810220 (+ 219us) server_negotiation.cc:407] Connection header received
0925 02:57:52.810409 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:52.810415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:52.810479 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:52.810606 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:52.811514 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:52.812243 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:52.813453 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:52.813858 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:52.813988 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:52.826722 (+ 12734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.194:43807: BlockingRecv error: recv got EOF from 127.18.196.194:43807 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 02:57:53 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:53 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:53.238101 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:53.230067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.195:57281)
0925 02:57:53.230634 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:53.230648 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:53.230762 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:53.231101 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:53.231105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:53.231482 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:53.231637 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:53.231646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:53.232614 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:53.232618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:53.233512 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:53.233523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:53.234361 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:53.234365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:53.234518 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:53.235198 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:53.235215 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:53.237872 (+ 2657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":325,"threads_started":1}
W20260925 02:57:53.238266 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:53.230106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:57281 (local address 127.18.196.254:40393)
0925 02:57:53.230301 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:53.230306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:53.230731 (+ 425us) server_negotiation.cc:407] Connection header received
0925 02:57:53.230891 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:53.230900 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:53.230974 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:53.231053 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:53.231798 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:53.232494 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:53.233636 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:53.234111 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:53.234204 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:53.238063 (+ 3859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.195:57281: BlockingRecv error: recv got EOF from 127.18.196.195:57281 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 02:57:53 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:53 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:53.368368 28573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:53.359925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.193:46307)
0925 02:57:53.360447 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:53.360462 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:53.360620 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:53.360978 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:53.360982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:53.361347 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:53.361565 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:53.361574 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:53.362676 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:53.362681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:53.363691 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:53.363702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:53.364449 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:53.364456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:53.364668 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:53.365213 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:53.365241 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:53.367985 (+ 2744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":261,"threads_started":1}
W20260925 02:57:53.368513 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:53.360025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:46307 (local address 127.18.196.254:40393)
0925 02:57:53.360355 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:53.360361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:53.360548 (+ 187us) server_negotiation.cc:407] Connection header received
0925 02:57:53.360743 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:53.360748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:53.360833 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:53.360957 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:53.361716 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:53.362514 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:53.363844 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:53.364284 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:53.364420 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:53.368249 (+ 3829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.193:46307: BlockingRecv error: recv got EOF from 127.18.196.193:46307 (error 108)
Metrics: {"server-negotiator.queue_time_us":186}
Sep 25 02:57:53 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:53 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:53.836436 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:53.827866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.194:47539)
0925 02:57:53.828371 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:53.828400 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:53.828555 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:53.828969 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:53.828974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:53.829341 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:53.829608 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:53.829620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:53.830616 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:53.830620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:53.831585 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:53.831597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:53.832285 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:53.832289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:53.832402 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:53.833084 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:53.833101 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:53.836199 (+ 3098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":278,"threads_started":1}
W20260925 02:57:53.836613 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:53.828093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:47539 (local address 127.18.196.254:40393)
0925 02:57:53.828244 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:53.828249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:53.828523 (+ 274us) server_negotiation.cc:407] Connection header received
0925 02:57:53.828702 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:53.828707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:53.828790 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:53.828895 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:53.829763 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:53.830495 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:53.831724 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:53.832160 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:53.832255 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:53.836393 (+ 4138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.194:47539: BlockingRecv error: recv got EOF from 127.18.196.194:47539 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 02:57:54 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:54 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.195@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:54.247762 28575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:54.239110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.195:45299)
0925 02:57:54.239571 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:54.239587 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:54.239751 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:54.240115 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:54.240120 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:54.240519 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:54.240693 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:54.240710 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:54.241755 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:54.241772 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:54.242697 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:54.242709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:54.243495 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:54.243501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:54.243686 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:54.244268 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:54.244284 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:54.247534 (+ 3250us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":239,"threads_started":1}
W20260925 02:57:54.248031 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:54.239163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:45299 (local address 127.18.196.254:40393)
0925 02:57:54.239382 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:54.239390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:54.239677 (+ 287us) server_negotiation.cc:407] Connection header received
0925 02:57:54.239901 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:54.239911 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:54.239973 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:54.240081 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:54.240855 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:54.241596 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:54.242896 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:54.243327 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:54.243479 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:54.247753 (+ 4274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.195:45299: BlockingRecv error: recv got EOF from 127.18.196.195:45299 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 02:57:54 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:54 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.193@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:54.378266 28576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:54.369641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.193:52219)
0925 02:57:54.370164 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:54.370180 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:54.370351 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:54.370742 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:54.370747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:54.371188 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:54.371352 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:54.371361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:54.372381 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:54.372387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:54.373348 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:54.373359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:54.374120 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:54.374126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:54.374300 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:54.374921 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:54.374938 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:54.377944 (+ 3006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":271,"threads_started":1}
W20260925 02:57:54.378392 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:54.369731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:52219 (local address 127.18.196.254:40393)
0925 02:57:54.369962 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:54.369972 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:54.370269 (+ 297us) server_negotiation.cc:407] Connection header received
0925 02:57:54.370504 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:54.370510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:54.370597 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:54.370708 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:54.371502 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:54.372237 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:54.373500 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:54.373926 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:54.374088 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:54.378139 (+ 4051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.193:52219: BlockingRecv error: recv got EOF from 127.18.196.193:52219 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 25 02:57:54 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:57:54 dist-test-slave-8zvk krb5kdc[27524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.196.194@KRBTEST.COM for oryx/127.18.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:57:54.846715 28577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:54.837671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40393 (local address 127.18.196.194:42807)
0925 02:57:54.838207 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:54.838225 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:54.838394 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:54.838750 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:54.838755 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:54.839346 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:54.839523 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:54.839533 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:54.840947 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:54.840951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:54.841980 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:54.841993 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:54.842722 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:54.842726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:54.842852 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:54.843435 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:54.843452 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:54.846463 (+ 3011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.196.254:40393: Server oryx/127.18.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":270,"threads_started":1}
W20260925 02:57:54.846908 28267 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:54.837746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:42807 (local address 127.18.196.254:40393)
0925 02:57:54.838001 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:54.838020 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:54.838323 (+ 303us) server_negotiation.cc:407] Connection header received
0925 02:57:54.838531 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:54.838538 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:54.838609 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:54.838717 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:54.839687 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:54.840793 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:54.842140 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:54.842582 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:54.842683 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:54.846661 (+ 3978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.196.194:42807: BlockingRecv error: recv got EOF from 127.18.196.194:42807 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
I20260925 02:57:55.099354 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 28116
I20260925 02:57:55.111083 28579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:55.100207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54082 (local address 127.18.196.193:40783)
0925 02:57:55.100773 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:55.100780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:55.100808 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:57:55.100864 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:55.100871 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:55.101088 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:55.101227 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:55.102741 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:55.103835 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:55.105059 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:55.105606 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:55.105697 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:55.107998 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:55.108006 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:55.108018 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:55.108055 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:55.110553 (+ 2498us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.18.196.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 02:57:55.110791 (+ 238us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54082: Request ticket server kudu/127.18.196.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":295,"threads_started":1}
W20260925 02:57:55.111158 28579 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54082: Request ticket server kudu/127.18.196.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 02:57:55.111455 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 28116: Not authorized: Client connection negotiation failed: client connection to 127.18.196.193:40783: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.18.196.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 02:57:55.111632 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 28116
I20260925 02:57:55.128034 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 28269
I20260925 02:57:55.139575 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:55.128891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55052 (local address 127.18.196.194:33749)
0925 02:57:55.129474 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:55.129481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:55.129510 (+ 29us) server_negotiation.cc:407] Connection header received
0925 02:57:55.129574 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:55.129581 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:55.129799 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:55.129974 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:55.131196 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:55.132284 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:55.133414 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:55.133965 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:55.134072 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:55.136386 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:55.136394 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:55.136423 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:55.136455 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:55.139032 (+ 2577us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.18.196.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 02:57:55.139267 (+ 235us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55052: Request ticket server kudu/127.18.196.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":263,"threads_started":1}
W20260925 02:57:55.139684 28580 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55052: Request ticket server kudu/127.18.196.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 02:57:55.139674 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 28269: Not authorized: Client connection negotiation failed: client connection to 127.18.196.194:33749: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.18.196.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 02:57:55.139824 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 28269
I20260925 02:57:55.156586 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 28401
W20260925 02:57:55.168777 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 28401: Not authorized: Client connection negotiation failed: client connection to 127.18.196.195:43967: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.18.196.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 02:57:55.168809 28581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:55.157669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49464 (local address 127.18.196.195:43967)
0925 02:57:55.158219 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:55.158226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:55.158251 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:57:55.158309 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:55.158315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:55.158523 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:55.158672 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:55.160134 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:55.161113 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:55.162453 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:55.163089 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:55.163182 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:55.165462 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:55.165469 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:55.165482 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:55.165516 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:55.168167 (+ 2651us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.18.196.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 02:57:55.168505 (+ 338us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49464: Request ticket server kudu/127.18.196.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":289,"threads_started":1}
I20260925 02:57:55.168938 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 28401
W20260925 02:57:55.168941 28581 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49464: Request ticket server kudu/127.18.196.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 02:57:55.185777 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 28047
I20260925 02:57:55.187045 28109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:55.305070 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 28047
2026-09-25T02:57:55Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20915 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T02:57:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:57:55Z Disabled control of system clock
I20260925 02:57:55.377609 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:40887
--webserver_interface=127.18.196.254
--webserver_port=0
--builtin_ntp_servers=127.18.196.212:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:40887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 02:57:55.693926 28586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:55.694195 28586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:55.694322 28586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:55.701427 28586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:55.701530 28586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:55.701570 28586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:55.701627 28586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:55.701669 28586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:55.710548 28586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:40887
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:40887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:55.711807 28586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:55.713474 28586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:55.725617 28592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:55.725657 28594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:55.725697 28591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:55.726097 28586 server_base.cc:1034] running on GCE node
I20260925 02:57:55.726948 28586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:55.728295 28586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:55.729501 28586 hybrid_clock.cc:648] HybridClock initialized: now 1790305075729431 us; error 53 us; skew 500 ppm
I20260925 02:57:55.729862 28586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:57:55.732101 28586 webserver.cc:460] Webserver started at http://127.18.196.254:36351/ using document root <none> and password file <none>
I20260925 02:57:55.732810 28586 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:55.732935 28586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:55.733289 28586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:55.736222 28586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "1d61058658e744f58ec4497ce98d88ec"
format_stamp: "Formatted at 2026-09-25 02:57:55 on dist-test-slave-8zvk"
server_key: "06f15a4a09da52828c941af0a3550f6b"
server_key_iv: "e4a36d925735b2d1e5aeb5b76e9ba3e6"
server_key_version: "encryptionkey@0"
I20260925 02:57:55.737057 28586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "1d61058658e744f58ec4497ce98d88ec"
format_stamp: "Formatted at 2026-09-25 02:57:55 on dist-test-slave-8zvk"
server_key: "06f15a4a09da52828c941af0a3550f6b"
server_key_iv: "e4a36d925735b2d1e5aeb5b76e9ba3e6"
server_key_version: "encryptionkey@0"
I20260925 02:57:55.742079 28586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:57:55.745541 28600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:55.746956 28586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:57:55.747150 28586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "1d61058658e744f58ec4497ce98d88ec"
format_stamp: "Formatted at 2026-09-25 02:57:55 on dist-test-slave-8zvk"
server_key: "06f15a4a09da52828c941af0a3550f6b"
server_key_iv: "e4a36d925735b2d1e5aeb5b76e9ba3e6"
server_key_version: "encryptionkey@0"
I20260925 02:57:55.747372 28586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:55.767742 28586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:55.768570 28586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:55.768872 28586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:55.787140 28586 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:40887
I20260925 02:57:55.787165 28651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:40887 every 8 connection(s)
I20260925 02:57:55.788477 28586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:57:55.792023 28652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:55.796182 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 28586
I20260925 02:57:55.796396 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:57:55.797048 19219 external_mini_cluster.cc:1428] Setting key 2cdb706023f078a8a6be30da897f2541
I20260925 02:57:55.804243 28652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d61058658e744f58ec4497ce98d88ec: Bootstrap starting.
I20260925 02:57:55.807912 28652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d61058658e744f58ec4497ce98d88ec: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:55.809286 28652 log.cc:824] T 00000000000000000000000000000000 P 1d61058658e744f58ec4497ce98d88ec: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:55.812616 28652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d61058658e744f58ec4497ce98d88ec: No bootstrap required, opened a new log
I20260925 02:57:55.812821 28655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:55.799106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36376 (local address 127.18.196.254:40887)
0925 02:57:55.800161 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:55.800185 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:55.800244 (+ 59us) server_negotiation.cc:407] Connection header received
0925 02:57:55.802693 (+ 2449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:55.802721 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:55.803285 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:55.803688 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:55.804197 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:55.806178 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:55.807658 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:55.808722 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:55.808909 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:55.810149 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:55.810159 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:55.810173 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:55.810814 (+ 641us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:55.810940 (+ 126us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:55.810966 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:55.811074 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:55.811288 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 02:57:55.811615 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":321,"threads_started":1}
I20260925 02:57:55.820348 28652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d61058658e744f58ec4497ce98d88ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d61058658e744f58ec4497ce98d88ec" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40887 } }
I20260925 02:57:55.820679 28652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d61058658e744f58ec4497ce98d88ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:55.820742 28652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d61058658e744f58ec4497ce98d88ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d61058658e744f58ec4497ce98d88ec, State: Initialized, Role: FOLLOWER
I20260925 02:57:55.821197 28652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d61058658e744f58ec4497ce98d88ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d61058658e744f58ec4497ce98d88ec" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40887 } }
I20260925 02:57:55.821470 28652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d61058658e744f58ec4497ce98d88ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:57:55.821653 28652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d61058658e744f58ec4497ce98d88ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:57:55.821844 28652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d61058658e744f58ec4497ce98d88ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:55.824034 28652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d61058658e744f58ec4497ce98d88ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d61058658e744f58ec4497ce98d88ec" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40887 } }
I20260925 02:57:55.824512 28652 leader_election.cc:302] T 00000000000000000000000000000000 P 1d61058658e744f58ec4497ce98d88ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d61058658e744f58ec4497ce98d88ec; no voters:
I20260925 02:57:55.825088 28652 leader_election.cc:288] T 00000000000000000000000000000000 P 1d61058658e744f58ec4497ce98d88ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:57:55.825209 28657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d61058658e744f58ec4497ce98d88ec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:55.826400 28657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d61058658e744f58ec4497ce98d88ec [term 1 LEADER]: Becoming Leader. State: Replica: 1d61058658e744f58ec4497ce98d88ec, State: Running, Role: LEADER
I20260925 02:57:55.827049 28657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d61058658e744f58ec4497ce98d88ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d61058658e744f58ec4497ce98d88ec" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40887 } }
I20260925 02:57:55.827252 28652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d61058658e744f58ec4497ce98d88ec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:55.829806 28659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d61058658e744f58ec4497ce98d88ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d61058658e744f58ec4497ce98d88ec. Latest consensus state: current_term: 1 leader_uuid: "1d61058658e744f58ec4497ce98d88ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d61058658e744f58ec4497ce98d88ec" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40887 } } }
I20260925 02:57:55.830237 28659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d61058658e744f58ec4497ce98d88ec [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:55.830881 28658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d61058658e744f58ec4497ce98d88ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d61058658e744f58ec4497ce98d88ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d61058658e744f58ec4497ce98d88ec" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40887 } } }
I20260925 02:57:55.831194 28658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d61058658e744f58ec4497ce98d88ec [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:55.831204 28666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:55.836496 28666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:55.844897 28666 catalog_manager.cc:1347] Generated new cluster ID: 4736b30e60f445778d3a247b868f8120
I20260925 02:57:55.844996 28666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:55.855684 28666 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:57:55.856839 28666 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:55.864836 28666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d61058658e744f58ec4497ce98d88ec: Generated new TSK 0
I20260925 02:57:55.865550 28666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:57:55.877166 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:40887
--builtin_ntp_servers=127.18.196.212:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 02:57:56.227612 28676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:56.227941 28676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:56.228108 28676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:56.236279 28676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:56.236382 28676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:56.236533 28676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 02:57:56.245739 28676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:40887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:56.247114 28676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:56.248888 28676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:56.262285 28682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:56.262362 28684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:56.262305 28681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:56.263242 28676 server_base.cc:1034] running on GCE node
I20260925 02:57:56.263844 28676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:56.264837 28676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:56.266075 28676 hybrid_clock.cc:648] HybridClock initialized: now 1790305076266043 us; error 57 us; skew 500 ppm
I20260925 02:57:56.266405 28676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:57:56.268661 28676 webserver.cc:460] Webserver started at http://127.18.196.193:39565/ using document root <none> and password file <none>
I20260925 02:57:56.269429 28676 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:56.269559 28676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:56.269970 28676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:56.273211 28676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "48e73192b68042ae99d8ce256f37dd0e"
format_stamp: "Formatted at 2026-09-25 02:57:56 on dist-test-slave-8zvk"
server_key: "e43f304294333b1d2d7b1795a84aa82b"
server_key_iv: "a66ca92745733c42013eac9ebfdd4ce5"
server_key_version: "encryptionkey@0"
I20260925 02:57:56.273958 28676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "48e73192b68042ae99d8ce256f37dd0e"
format_stamp: "Formatted at 2026-09-25 02:57:56 on dist-test-slave-8zvk"
server_key: "e43f304294333b1d2d7b1795a84aa82b"
server_key_iv: "a66ca92745733c42013eac9ebfdd4ce5"
server_key_version: "encryptionkey@0"
I20260925 02:57:56.279119 28676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260925 02:57:56.282493 28690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:56.283896 28676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:57:56.284085 28676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "48e73192b68042ae99d8ce256f37dd0e"
format_stamp: "Formatted at 2026-09-25 02:57:56 on dist-test-slave-8zvk"
server_key: "e43f304294333b1d2d7b1795a84aa82b"
server_key_iv: "a66ca92745733c42013eac9ebfdd4ce5"
server_key_version: "encryptionkey@0"
I20260925 02:57:56.284322 28676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:56.303176 28676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:56.304044 28676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:56.304368 28676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:56.305368 28676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:56.306952 28676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:56.307085 28676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:56.307225 28676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:56.307323 28676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:56.341861 28676 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:34541
I20260925 02:57:56.341956 28802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:34541 every 8 connection(s)
I20260925 02:57:56.343309 28676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 02:57:56.353796 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 28676
I20260925 02:57:56.354008 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 02:57:56.354882 19219 external_mini_cluster.cc:1428] Setting key ce151a68be19113707513dbf82608201
I20260925 02:57:56.356498 28805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:56.345991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:56621 (local address 127.18.196.254:40887)
0925 02:57:56.346471 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:56.346476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:56.347711 (+ 1235us) server_negotiation.cc:407] Connection header received
0925 02:57:56.349529 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:56.349533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:56.349599 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:56.349695 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:56.351007 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:56.351759 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:56.353316 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:56.353702 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:56.353793 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:56.355563 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:56.355568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:56.355571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:56.355589 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:56.355632 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:56.355643 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:56.355766 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:56.356034 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 02:57:56.356129 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":230,"threads_started":1}
I20260925 02:57:56.357594 28806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:56.346561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40887 (local address 127.18.196.193:56621)
0925 02:57:56.347562 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:56.347599 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:56.349183 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:56.350049 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:56.350062 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:56.350163 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:56.350769 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:56.350787 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:56.351943 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:56.351947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:56.353183 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:56.353200 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:56.353945 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:56.353954 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:56.354281 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:56.355202 (+ 921us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:57:56.355205 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:56.355257 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:57:56.355357 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:56.355766 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:56.355781 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:56.355791 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 02:57:56.355983 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 02:57:56.356387 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":339,"threads_started":1}
I20260925 02:57:56.357962 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:40887
--builtin_ntp_servers=127.18.196.212:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 02:57:56.359304 28803 heartbeater.cc:347] Connected to a master server at 127.18.196.254:40887
I20260925 02:57:56.359647 28803 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:56.360601 28803 heartbeater.cc:511] Master 127.18.196.254:40887 requested a full tablet report, sending...
I20260925 02:57:56.362962 28617 ts_manager.cc:194] Registered new tserver with Master: 48e73192b68042ae99d8ce256f37dd0e (127.18.196.193:34541)
I20260925 02:57:56.365199 28617 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.196.193:56621
W20260925 02:57:56.688397 28807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:56.688745 28807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:56.688871 28807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:56.696034 28807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:56.696125 28807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:56.696224 28807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 02:57:56.705161 28807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:40887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:56.706372 28807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:56.708112 28807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:56.720526 28813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:56.720526 28812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:56.721881 28815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:56.722368 28807 server_base.cc:1034] running on GCE node
I20260925 02:57:56.723043 28807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:56.723945 28807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:56.725189 28807 hybrid_clock.cc:648] HybridClock initialized: now 1790305076725145 us; error 55 us; skew 500 ppm
I20260925 02:57:56.725548 28807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:57:56.727919 28807 webserver.cc:460] Webserver started at http://127.18.196.194:36105/ using document root <none> and password file <none>
I20260925 02:57:56.728672 28807 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:56.728812 28807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:56.729152 28807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:56.732240 28807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "e0046046abf54a21868afd56bc21e56b"
format_stamp: "Formatted at 2026-09-25 02:57:56 on dist-test-slave-8zvk"
server_key: "bf73546936b52dd5a860efab1f32f01a"
server_key_iv: "b050e3d9e726e419b470dc748cb72dbb"
server_key_version: "encryptionkey@0"
I20260925 02:57:56.733068 28807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "e0046046abf54a21868afd56bc21e56b"
format_stamp: "Formatted at 2026-09-25 02:57:56 on dist-test-slave-8zvk"
server_key: "bf73546936b52dd5a860efab1f32f01a"
server_key_iv: "b050e3d9e726e419b470dc748cb72dbb"
server_key_version: "encryptionkey@0"
I20260925 02:57:56.738108 28807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 02:57:56.741777 28821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:56.743170 28807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:57:56.743450 28807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "e0046046abf54a21868afd56bc21e56b"
format_stamp: "Formatted at 2026-09-25 02:57:56 on dist-test-slave-8zvk"
server_key: "bf73546936b52dd5a860efab1f32f01a"
server_key_iv: "b050e3d9e726e419b470dc748cb72dbb"
server_key_version: "encryptionkey@0"
I20260925 02:57:56.743676 28807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:56.757961 28807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:56.758790 28807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:56.759140 28807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:56.760047 28807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:56.761480 28807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:56.761610 28807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:56.761754 28807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:56.761855 28807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:56.796555 28807 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:38155
I20260925 02:57:56.796603 28933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:38155 every 8 connection(s)
I20260925 02:57:56.797847 28807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 02:57:56.803725 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 28807
I20260925 02:57:56.803894 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 02:57:56.804445 19219 external_mini_cluster.cc:1428] Setting key 95597e431c9f07ff824ac5813518da30
I20260925 02:57:56.807498 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:40887
--builtin_ntp_servers=127.18.196.212:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 02:57:56.812568 28805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:56.800611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:38311 (local address 127.18.196.254:40887)
0925 02:57:56.800790 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:56.800795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:56.802329 (+ 1534us) server_negotiation.cc:407] Connection header received
0925 02:57:56.804732 (+ 2403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:56.804741 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:56.804828 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:56.804918 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:56.806162 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:56.807069 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:56.809677 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:56.810057 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:56.810161 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:56.811839 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:56.811844 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:56.811847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:56.811866 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:56.811898 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:56.811910 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:56.811980 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:56.812248 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 02:57:56.812329 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:57:56.814134 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:56.801221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40887 (local address 127.18.196.194:38311)
0925 02:57:56.802161 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:56.802200 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:56.804347 (+ 2147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:56.805203 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:56.805212 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:56.805297 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:56.805991 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:56.806010 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:56.807229 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:56.807238 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:56.809506 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:56.809528 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:56.810303 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:56.810312 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:56.810652 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:56.811472 (+ 820us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:57:56.811479 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:56.811537 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:57:56.811643 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:56.812018 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:56.812031 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:56.812042 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 02:57:56.812233 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 02:57:56.812688 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":338,"threads_started":1}
I20260925 02:57:56.815613 28934 heartbeater.cc:347] Connected to a master server at 127.18.196.254:40887
I20260925 02:57:56.815955 28934 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:56.816833 28934 heartbeater.cc:511] Master 127.18.196.254:40887 requested a full tablet report, sending...
I20260925 02:57:56.818375 28617 ts_manager.cc:194] Registered new tserver with Master: e0046046abf54a21868afd56bc21e56b (127.18.196.194:38155)
I20260925 02:57:56.819602 28617 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.196.194:38311
W20260925 02:57:57.140010 28937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:57.140277 28937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:57.140364 28937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:57.147396 28937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:57.147660 28937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:57.147873 28937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 02:57:57.157050 28937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:40887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:57.158324 28937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:57.160109 28937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:57.173151 28942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:57.173125 28943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:57.173431 28945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:57.173977 28937 server_base.cc:1034] running on GCE node
I20260925 02:57:57.174578 28937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:57.175578 28937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:57.176821 28937 hybrid_clock.cc:648] HybridClock initialized: now 1790305077176784 us; error 56 us; skew 500 ppm
I20260925 02:57:57.177142 28937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:57:57.179353 28937 webserver.cc:460] Webserver started at http://127.18.196.195:33551/ using document root <none> and password file <none>
I20260925 02:57:57.180047 28937 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:57.180182 28937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:57.180506 28937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:57.183372 28937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "e49891014a384687b1ee70f09f18c4f7"
format_stamp: "Formatted at 2026-09-25 02:57:57 on dist-test-slave-8zvk"
server_key: "23dd247431d4a7489d837d9699902307"
server_key_iv: "3197311a104dcca78c8fddef1c938972"
server_key_version: "encryptionkey@0"
I20260925 02:57:57.184188 28937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "e49891014a384687b1ee70f09f18c4f7"
format_stamp: "Formatted at 2026-09-25 02:57:57 on dist-test-slave-8zvk"
server_key: "23dd247431d4a7489d837d9699902307"
server_key_iv: "3197311a104dcca78c8fddef1c938972"
server_key_version: "encryptionkey@0"
I20260925 02:57:57.189252 28937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:57:57.192633 28951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:57.193912 28937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:57:57.194094 28937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "e49891014a384687b1ee70f09f18c4f7"
format_stamp: "Formatted at 2026-09-25 02:57:57 on dist-test-slave-8zvk"
server_key: "23dd247431d4a7489d837d9699902307"
server_key_iv: "3197311a104dcca78c8fddef1c938972"
server_key_version: "encryptionkey@0"
I20260925 02:57:57.194339 28937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:57.207829 28937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:57.208595 28937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:57.208894 28937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:57.209800 28937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:57.211272 28937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:57.211405 28937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:57.211540 28937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:57.211638 28937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:57.249907 28937 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:33049
I20260925 02:57:57.249951 29063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:33049 every 8 connection(s)
I20260925 02:57:57.251364 28937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 02:57:57.253229 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 28937
I20260925 02:57:57.253399 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 02:57:57.253924 19219 external_mini_cluster.cc:1428] Setting key 09f70e5e1bfe8d62b7a957bcb3ba092d
I20260925 02:57:57.264736 28805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:57.254056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:34631 (local address 127.18.196.254:40887)
0925 02:57:57.254263 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:57.254269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:57.256027 (+ 1758us) server_negotiation.cc:407] Connection header received
0925 02:57:57.258188 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:57.258193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:57.258276 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:57.258398 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:57.259631 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:57.260367 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:57.261911 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:57.262309 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:57.262444 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:57.263839 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:57.263844 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:57.263847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:57.263865 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:57.263895 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:57.263907 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:57.264030 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:57.264270 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 02:57:57.264388 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 02:57:57.265657 29066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:57.254751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:40887 (local address 127.18.196.195:34631)
0925 02:57:57.255882 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:57.255931 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:57.257915 (+ 1984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:57.258649 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:57.258658 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:57.258745 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:57.259464 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:57.259483 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:57.260523 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:57.260528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:57.261758 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:57.261771 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:57.262556 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:57.262565 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:57.262776 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:57.263459 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:57:57.263462 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:57.263526 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:57:57.263613 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:57.264032 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:57.264046 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:57.264054 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 02:57:57.264242 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 02:57:57.264622 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":316,"threads_started":1}
I20260925 02:57:57.267102 29064 heartbeater.cc:347] Connected to a master server at 127.18.196.254:40887
I20260925 02:57:57.267424 29064 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:57.268194 29064 heartbeater.cc:511] Master 127.18.196.254:40887 requested a full tablet report, sending...
I20260925 02:57:57.269520 28617 ts_manager.cc:194] Registered new tserver with Master: e49891014a384687b1ee70f09f18c4f7 (127.18.196.195:33049)
I20260925 02:57:57.270298 28617 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.196.195:34631
I20260925 02:57:57.281689 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 02:57:57.287951 28805 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:57.286279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36380 (local address 127.18.196.254:40887)
0925 02:57:57.286549 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:57.286555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:57.286897 (+ 342us) server_negotiation.cc:407] Connection header received
0925 02:57:57.287044 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:57.287048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:57.287187 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:57.287269 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:57.287745 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36380: BlockingRecv error: recv got EOF from 127.0.0.1:36380 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 02:57:57.288897 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 28676
I20260925 02:57:57.296748 29073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:57.289903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34406 (local address 127.18.196.193:34541)
0925 02:57:57.290422 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:57.290431 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:57.290459 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:57:57.290519 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:57.290524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:57.290722 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:57.290909 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:57.291349 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:57.292303 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:57.293562 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:57.294152 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:57.294248 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:57.295329 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:57.295337 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:57.295349 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:57.295920 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:57.295966 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:57.295986 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:57.296090 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:57.296238 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 02:57:57.296419 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":249,"threads_started":1}
I20260925 02:57:57.298066 28798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:57.375949 28803 heartbeater.cc:503] Master 127.18.196.254:40887 was elected leader, sending a full tablet report...
I20260925 02:57:57.377761 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 28676
I20260925 02:57:57.394236 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 28807
I20260925 02:57:57.402055 29075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:57.395132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44060 (local address 127.18.196.194:38155)
0925 02:57:57.395645 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:57.395668 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:57.395693 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:57:57.395750 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:57.395756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:57.395980 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:57.396156 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:57.396555 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:57.397541 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:57.398788 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:57.399495 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:57.399633 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:57.400641 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:57.400649 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:57.400661 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:57.401257 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:57.401302 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:57.401322 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:57.401497 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:57.401592 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 02:57:57.401717 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":260,"threads_started":1}
I20260925 02:57:57.403347 28929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:57.483727 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 28807
I20260925 02:57:57.500702 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 28937
I20260925 02:57:57.508287 29077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:57.501607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51350 (local address 127.18.196.195:33049)
0925 02:57:57.502235 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:57.502248 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:57.502279 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:57:57.502340 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:57.502346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:57.502568 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:57.502696 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:57.503186 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:57.504197 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:57.505353 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:57.505869 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:57.505967 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:57.506991 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:57.507000 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:57.507013 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:57.507582 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:57.507627 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:57.507648 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:57.507747 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:57.507880 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 02:57:57.507982 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":328,"threads_started":1}
I20260925 02:57:57.509536 29059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:57.591936 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 28937
I20260925 02:57:57.608095 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 28586
I20260925 02:57:57.609274 28647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:57.709088 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 28586
2026-09-25T02:57:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2399 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T02:57:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:57:57Z Disabled control of system clock
I20260925 02:57:57.780813 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:39165
--webserver_interface=127.18.196.254
--webserver_port=0
--builtin_ntp_servers=127.18.196.212:44451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:39165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 02:57:58.100755 29083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:58.101019 29083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:58.101099 29083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:58.108187 29083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:58.108282 29083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:58.108321 29083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:58.108381 29083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:58.108417 29083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:58.117131 29083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:44451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:39165
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:39165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:58.118224 29083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:58.119977 29083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:58.131722 29089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:58.131763 29091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:58.131876 29088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:58.132926 29083 server_base.cc:1034] running on GCE node
I20260925 02:57:58.133668 29083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:58.135030 29083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:58.136256 29083 hybrid_clock.cc:648] HybridClock initialized: now 1790305078136210 us; error 53 us; skew 500 ppm
I20260925 02:57:58.136598 29083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:57:58.144418 29083 webserver.cc:460] Webserver started at http://127.18.196.254:35375/ using document root <none> and password file <none>
I20260925 02:57:58.145116 29083 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:58.145253 29083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:58.145573 29083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:58.150072 29083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "313a0d098091492cb976ab5cd2292219"
format_stamp: "Formatted at 2026-09-25 02:57:58 on dist-test-slave-8zvk"
server_key: "2b5c21c2263c85f2693d6b1f675ac841"
server_key_iv: "6f1bc22c9c2005c759a99b38ab2781c9"
server_key_version: "encryptionkey@0"
I20260925 02:57:58.150923 29083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "313a0d098091492cb976ab5cd2292219"
format_stamp: "Formatted at 2026-09-25 02:57:58 on dist-test-slave-8zvk"
server_key: "2b5c21c2263c85f2693d6b1f675ac841"
server_key_iv: "6f1bc22c9c2005c759a99b38ab2781c9"
server_key_version: "encryptionkey@0"
I20260925 02:57:58.155859 29083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260925 02:57:58.159192 29097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:58.160674 29083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 02:57:58.160880 29083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "313a0d098091492cb976ab5cd2292219"
format_stamp: "Formatted at 2026-09-25 02:57:58 on dist-test-slave-8zvk"
server_key: "2b5c21c2263c85f2693d6b1f675ac841"
server_key_iv: "6f1bc22c9c2005c759a99b38ab2781c9"
server_key_version: "encryptionkey@0"
I20260925 02:57:58.161113 29083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:58.166517 29083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:58.167305 29083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:58.167608 29083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:58.185127 29083 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:39165
I20260925 02:57:58.185168 29148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:39165 every 8 connection(s)
I20260925 02:57:58.186352 29083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:57:58.189929 29149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:58.190274 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 29083
I20260925 02:57:58.190485 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:57:58.191315 19219 external_mini_cluster.cc:1428] Setting key 01760be80c16afd8431741354d70e26b
I20260925 02:57:58.200728 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:58.193310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.18.196.254:39165)
0925 02:57:58.194315 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:58.194335 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:58.194383 (+ 48us) server_negotiation.cc:407] Connection header received
0925 02:57:58.196949 (+ 2566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:58.196973 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:58.197477 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:58.197890 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:58.198888 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:58.198896 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:58.198911 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:58.198927 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:58.199005 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:58.199026 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:58.199223 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:58.199359 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 02:57:58.199593 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":341,"threads_started":1}
I20260925 02:57:58.204468 29149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 313a0d098091492cb976ab5cd2292219: Bootstrap starting.
I20260925 02:57:58.207402 29149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 313a0d098091492cb976ab5cd2292219: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:58.208293 29149 log.cc:824] T 00000000000000000000000000000000 P 313a0d098091492cb976ab5cd2292219: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:58.210824 29149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 313a0d098091492cb976ab5cd2292219: No bootstrap required, opened a new log
I20260925 02:57:58.218546 29149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 313a0d098091492cb976ab5cd2292219 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313a0d098091492cb976ab5cd2292219" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39165 } }
I20260925 02:57:58.218889 29149 raft_consensus.cc:374] T 00000000000000000000000000000000 P 313a0d098091492cb976ab5cd2292219 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:58.218962 29149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 313a0d098091492cb976ab5cd2292219 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313a0d098091492cb976ab5cd2292219, State: Initialized, Role: FOLLOWER
I20260925 02:57:58.219532 29149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 313a0d098091492cb976ab5cd2292219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313a0d098091492cb976ab5cd2292219" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39165 } }
I20260925 02:57:58.219724 29149 raft_consensus.cc:388] T 00000000000000000000000000000000 P 313a0d098091492cb976ab5cd2292219 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:57:58.219817 29149 raft_consensus.cc:482] T 00000000000000000000000000000000 P 313a0d098091492cb976ab5cd2292219 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:57:58.219933 29149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 313a0d098091492cb976ab5cd2292219 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:58.221921 29149 raft_consensus.cc:504] T 00000000000000000000000000000000 P 313a0d098091492cb976ab5cd2292219 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313a0d098091492cb976ab5cd2292219" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39165 } }
I20260925 02:57:58.222326 29149 leader_election.cc:302] T 00000000000000000000000000000000 P 313a0d098091492cb976ab5cd2292219 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 313a0d098091492cb976ab5cd2292219; no voters:
I20260925 02:57:58.222886 29149 leader_election.cc:288] T 00000000000000000000000000000000 P 313a0d098091492cb976ab5cd2292219 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:57:58.223006 29154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 313a0d098091492cb976ab5cd2292219 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:58.223731 29154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 313a0d098091492cb976ab5cd2292219 [term 1 LEADER]: Becoming Leader. State: Replica: 313a0d098091492cb976ab5cd2292219, State: Running, Role: LEADER
I20260925 02:57:58.224270 29154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 313a0d098091492cb976ab5cd2292219 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313a0d098091492cb976ab5cd2292219" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39165 } }
I20260925 02:57:58.224893 29149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 313a0d098091492cb976ab5cd2292219 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:58.226807 29155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 313a0d098091492cb976ab5cd2292219 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "313a0d098091492cb976ab5cd2292219" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313a0d098091492cb976ab5cd2292219" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39165 } } }
I20260925 02:57:58.227216 29155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 313a0d098091492cb976ab5cd2292219 [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:58.228681 29156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 313a0d098091492cb976ab5cd2292219 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 313a0d098091492cb976ab5cd2292219. Latest consensus state: current_term: 1 leader_uuid: "313a0d098091492cb976ab5cd2292219" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313a0d098091492cb976ab5cd2292219" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39165 } } }
I20260925 02:57:58.229120 29156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 313a0d098091492cb976ab5cd2292219 [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:58.230264 29163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:58.235384 29163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:58.243539 29163 catalog_manager.cc:1347] Generated new cluster ID: 2b0d4c7e6d5649ef83a9b10c4891773a
I20260925 02:57:58.243650 29163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:58.276363 29163 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:57:58.276772 29163 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:58.288430 29163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 313a0d098091492cb976ab5cd2292219: Generated new TSK 0
I20260925 02:57:58.289211 29163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:57:58.296738 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:39165
--builtin_ntp_servers=127.18.196.212:44451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 02:57:58.636926 29173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:58.637173 29173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:58.637259 29173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:58.644337 29173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:58.644441 29173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:58.644546 29173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 02:57:58.653302 29173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:44451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:39165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:58.654433 29173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:58.656172 29173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:58.669130 29178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:58.669214 29179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:58.669320 29181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:58.670233 29173 server_base.cc:1034] running on GCE node
I20260925 02:57:58.671154 29173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:58.672176 29173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:58.673384 29173 hybrid_clock.cc:648] HybridClock initialized: now 1790305078673318 us; error 53 us; skew 500 ppm
I20260925 02:57:58.673735 29173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:57:58.675998 29173 webserver.cc:460] Webserver started at http://127.18.196.193:33767/ using document root <none> and password file <none>
I20260925 02:57:58.676685 29173 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:58.676800 29173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:58.677132 29173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:58.679948 29173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "c1f912c473914733927d2d0c22736a83"
format_stamp: "Formatted at 2026-09-25 02:57:58 on dist-test-slave-8zvk"
server_key: "12b7b479ba476b8f65100cf45ec58774"
server_key_iv: "2e1164dab82630268a774d410ff05ee8"
server_key_version: "encryptionkey@0"
I20260925 02:57:58.680747 29173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "c1f912c473914733927d2d0c22736a83"
format_stamp: "Formatted at 2026-09-25 02:57:58 on dist-test-slave-8zvk"
server_key: "12b7b479ba476b8f65100cf45ec58774"
server_key_iv: "2e1164dab82630268a774d410ff05ee8"
server_key_version: "encryptionkey@0"
I20260925 02:57:58.685796 29173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 02:57:58.689109 29187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:58.690469 29173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:57:58.690665 29173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "c1f912c473914733927d2d0c22736a83"
format_stamp: "Formatted at 2026-09-25 02:57:58 on dist-test-slave-8zvk"
server_key: "12b7b479ba476b8f65100cf45ec58774"
server_key_iv: "2e1164dab82630268a774d410ff05ee8"
server_key_version: "encryptionkey@0"
I20260925 02:57:58.690933 29173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:58.696692 29173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:58.697475 29173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:58.697749 29173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:58.698668 29173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:58.700268 29173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:58.700397 29173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:58.700533 29173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:58.700623 29173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:58.733835 29173 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:40687
I20260925 02:57:58.733888 29299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:40687 every 8 connection(s)
I20260925 02:57:58.735224 29173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 02:57:58.742763 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 29173
I20260925 02:57:58.743022 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 02:57:58.743264 29302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:58.737814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:36751 (local address 127.18.196.254:39165)
0925 02:57:58.738270 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:58.738274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:58.739358 (+ 1084us) server_negotiation.cc:407] Connection header received
0925 02:57:58.740991 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:58.740997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:58.741076 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:58.741194 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:58.742436 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:58.742441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:58.742446 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:58.742464 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:58.742514 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:58.742528 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:58.742663 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:58.742894 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 02:57:58.742964 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":253,"threads_started":1}
I20260925 02:57:58.743789 19219 external_mini_cluster.cc:1428] Setting key 389d9e53906d41a54f3a26de74efad5e
I20260925 02:57:58.744170 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:58.738287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39165 (local address 127.18.196.193:36751)
0925 02:57:58.739204 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:58.739248 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:58.740721 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:58.741434 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:58.741442 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:58.741550 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:58.742154 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:57:58.742156 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:58.742191 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:57:58.742264 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:58.742630 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:58.742643 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:58.742652 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 02:57:58.742972 (+ 320us) client_negotiation.cc:240] Negotiation successful
0925 02:57:58.743212 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":270,"threads_started":1}
I20260925 02:57:58.745728 29300 heartbeater.cc:347] Connected to a master server at 127.18.196.254:39165
I20260925 02:57:58.746008 29300 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:58.746529 29300 heartbeater.cc:511] Master 127.18.196.254:39165 requested a full tablet report, sending...
I20260925 02:57:58.746557 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:39165
--builtin_ntp_servers=127.18.196.212:44451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 02:57:58.748616 29114 ts_manager.cc:194] Registered new tserver with Master: c1f912c473914733927d2d0c22736a83 (127.18.196.193:40687)
W20260925 02:57:59.053802 29304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:59.054042 29304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:59.054123 29304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:59.060813 29304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:59.060897 29304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:59.061002 29304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 02:57:59.069542 29304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:44451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:39165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:59.070560 29304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:59.072162 29304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:59.084760 29309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:59.084726 29312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:59.084810 29310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:59.085424 29304 server_base.cc:1034] running on GCE node
I20260925 02:57:59.085979 29304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:59.086787 29304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:59.088054 29304 hybrid_clock.cc:648] HybridClock initialized: now 1790305079088007 us; error 35 us; skew 500 ppm
I20260925 02:57:59.088394 29304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:57:59.090744 29304 webserver.cc:460] Webserver started at http://127.18.196.194:42509/ using document root <none> and password file <none>
I20260925 02:57:59.091434 29304 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:59.091559 29304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:59.091903 29304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:59.094650 29304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "4178045413814c7a98a79406e3c9f998"
format_stamp: "Formatted at 2026-09-25 02:57:59 on dist-test-slave-8zvk"
server_key: "d68d8060a5706b7262f95f07ec2b9a46"
server_key_iv: "9cb551d719cd3290733c11832d74a6de"
server_key_version: "encryptionkey@0"
I20260925 02:57:59.095515 29304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "4178045413814c7a98a79406e3c9f998"
format_stamp: "Formatted at 2026-09-25 02:57:59 on dist-test-slave-8zvk"
server_key: "d68d8060a5706b7262f95f07ec2b9a46"
server_key_iv: "9cb551d719cd3290733c11832d74a6de"
server_key_version: "encryptionkey@0"
I20260925 02:57:59.100306 29304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:57:59.103592 29318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:59.104816 29304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:57:59.104990 29304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "4178045413814c7a98a79406e3c9f998"
format_stamp: "Formatted at 2026-09-25 02:57:59 on dist-test-slave-8zvk"
server_key: "d68d8060a5706b7262f95f07ec2b9a46"
server_key_iv: "9cb551d719cd3290733c11832d74a6de"
server_key_version: "encryptionkey@0"
I20260925 02:57:59.105165 29304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:59.110535 29304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:59.111302 29304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:59.111552 29304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:59.112421 29304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:59.113656 29304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:59.113760 29304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:59.113906 29304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:59.113999 29304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:59.146306 29304 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:33769
I20260925 02:57:59.146402 29430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:33769 every 8 connection(s)
I20260925 02:57:59.147575 29304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 02:57:59.148339 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 29304
I20260925 02:57:59.148558 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 02:57:59.149119 19219 external_mini_cluster.cc:1428] Setting key fca7aa4a8f5a415848d3752dc601b06c
I20260925 02:57:59.152088 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:39165
--builtin_ntp_servers=127.18.196.212:44451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 02:57:59.155648 29302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:59.150224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:38313 (local address 127.18.196.254:39165)
0925 02:57:59.150413 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:59.150418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:59.151768 (+ 1350us) server_negotiation.cc:407] Connection header received
0925 02:57:59.153453 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:59.153458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:59.153535 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:59.153650 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:59.154918 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:59.154924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:59.154929 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:59.154946 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:59.154992 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:59.155004 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:59.155124 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:59.155314 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 02:57:59.155382 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:57:59.156486 29433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:59.150651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39165 (local address 127.18.196.194:38313)
0925 02:57:59.151624 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:59.151657 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:59.153204 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:59.153869 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:59.153876 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:59.153960 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:59.154615 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:57:59.154618 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:59.154651 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:57:59.154730 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:59.155101 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:59.155112 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:59.155119 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 02:57:59.155390 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 02:57:59.155632 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":350,"threads_started":1}
I20260925 02:57:59.157713 29431 heartbeater.cc:347] Connected to a master server at 127.18.196.254:39165
I20260925 02:57:59.158013 29431 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:59.158550 29431 heartbeater.cc:511] Master 127.18.196.254:39165 requested a full tablet report, sending...
I20260925 02:57:59.159916 29114 ts_manager.cc:194] Registered new tserver with Master: 4178045413814c7a98a79406e3c9f998 (127.18.196.194:33769)
W20260925 02:57:59.472761 29434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:59.472981 29434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:59.473076 29434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:59.479691 29434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:59.479769 29434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:59.479878 29434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 02:57:59.488197 29434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:44451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:39165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:57:59.489260 29434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:59.490805 29434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:59.503468 29440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:59.503624 29439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:59.503643 29442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:59.504652 29434 server_base.cc:1034] running on GCE node
I20260925 02:57:59.505216 29434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:59.506062 29434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:59.507280 29434 hybrid_clock.cc:648] HybridClock initialized: now 1790305079507251 us; error 44 us; skew 500 ppm
I20260925 02:57:59.507591 29434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:57:59.509944 29434 webserver.cc:460] Webserver started at http://127.18.196.195:34461/ using document root <none> and password file <none>
I20260925 02:57:59.510560 29434 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:59.510644 29434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:59.510960 29434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:59.513511 29434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "2c3666858a2a4ab4aebb6dc6415970ea"
format_stamp: "Formatted at 2026-09-25 02:57:59 on dist-test-slave-8zvk"
server_key: "b322b0825f8cf4178b714e01e967cfc1"
server_key_iv: "8bac1f875b08548b7e83770be5b83343"
server_key_version: "encryptionkey@0"
I20260925 02:57:59.514206 29434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "2c3666858a2a4ab4aebb6dc6415970ea"
format_stamp: "Formatted at 2026-09-25 02:57:59 on dist-test-slave-8zvk"
server_key: "b322b0825f8cf4178b714e01e967cfc1"
server_key_iv: "8bac1f875b08548b7e83770be5b83343"
server_key_version: "encryptionkey@0"
I20260925 02:57:59.519021 29434 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260925 02:57:59.522219 29448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:59.523501 29434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:57:59.523664 29434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "2c3666858a2a4ab4aebb6dc6415970ea"
format_stamp: "Formatted at 2026-09-25 02:57:59 on dist-test-slave-8zvk"
server_key: "b322b0825f8cf4178b714e01e967cfc1"
server_key_iv: "8bac1f875b08548b7e83770be5b83343"
server_key_version: "encryptionkey@0"
I20260925 02:57:59.523835 29434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:59.529493 29434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:59.530200 29434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:59.530407 29434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:59.531288 29434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:57:59.532513 29434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:57:59.532608 29434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:59.532696 29434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:57:59.532747 29434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:59.564908 29434 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:46451
I20260925 02:57:59.564990 29560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:46451 every 8 connection(s)
I20260925 02:57:59.566207 29434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 02:57:59.572692 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 29434
I20260925 02:57:59.572887 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 02:57:59.573288 19219 external_mini_cluster.cc:1428] Setting key 99089aa875a6de3da15b642bc34de5eb
I20260925 02:57:59.574149 29302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:59.568688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:34901 (local address 127.18.196.254:39165)
0925 02:57:59.568900 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:59.568906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:59.570261 (+ 1355us) server_negotiation.cc:407] Connection header received
0925 02:57:59.571863 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:59.571867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:59.571925 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:59.572011 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:59.573353 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:59.573358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:59.573362 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:59.573397 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:59.573439 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:59.573454 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:59.573609 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:59.573798 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 02:57:59.573864 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 02:57:59.575146 29563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:59.569265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39165 (local address 127.18.196.195:34901)
0925 02:57:59.570128 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:59.570162 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:59.571633 (+ 1471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:59.572205 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:59.572212 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:59.572337 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:59.573086 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:57:59.573089 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:59.573121 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:57:59.573193 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:59.573583 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:59.573594 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:59.573601 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 02:57:59.573872 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 02:57:59.574115 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":267,"threads_started":1}
I20260925 02:57:59.576505 29561 heartbeater.cc:347] Connected to a master server at 127.18.196.254:39165
I20260925 02:57:59.576817 29561 heartbeater.cc:464] Registering TS with master...
I20260925 02:57:59.577359 29561 heartbeater.cc:511] Master 127.18.196.254:39165 requested a full tablet report, sending...
I20260925 02:57:59.578626 29114 ts_manager.cc:194] Registered new tserver with Master: 2c3666858a2a4ab4aebb6dc6415970ea (127.18.196.195:46451)
I20260925 02:57:59.588513 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 02:57:59.594512 29302 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:59.592777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41986 (local address 127.18.196.254:39165)
0925 02:57:59.592968 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:59.592977 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:59.593380 (+ 403us) server_negotiation.cc:407] Connection header received
0925 02:57:59.593697 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:59.593702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:59.593769 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:59.593906 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:59.594256 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41986: BlockingRecv error: recv got EOF from 127.0.0.1:41986 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:57:59.595468 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 29173
I20260925 02:57:59.599012 29570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:59.596553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42544 (local address 127.18.196.193:40687)
0925 02:57:59.597091 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:59.597097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:59.597121 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:57:59.597175 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:59.597180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:59.597360 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:59.597478 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:59.598292 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:59.598297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:59.598308 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:59.598319 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:59.598360 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:59.598373 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:59.598473 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:59.598555 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 02:57:59.598669 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":271,"threads_started":1}
I20260925 02:57:59.600096 29295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:59.678823 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 29173
I20260925 02:57:59.695557 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 29304
I20260925 02:57:59.699272 29572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:59.696487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56376 (local address 127.18.196.194:33769)
0925 02:57:59.696966 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:59.696979 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:59.697005 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:57:59.697065 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:59.697071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:59.697278 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:59.697421 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:59.698519 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:59.698526 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:59.698537 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:59.698557 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:59.698601 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:59.698616 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:59.698720 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:59.698801 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 02:57:59.698961 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":217,"threads_started":1}
I20260925 02:57:59.700568 29426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:59.781255 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 29304
I20260925 02:57:59.797567 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 29434
I20260925 02:57:59.801646 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:59.798665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60242 (local address 127.18.196.195:46451)
0925 02:57:59.799396 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:59.799403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:59.799433 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:57:59.799513 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:59.799520 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:59.799732 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:59.799860 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:59.800894 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:59.800900 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:59.800912 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:59.800924 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:59.800964 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:59.800979 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:59.801116 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:59.801166 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 02:57:59.801284 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":414,"threads_started":1}
I20260925 02:57:59.803052 29556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:57:59.884559 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 29434
I20260925 02:57:59.901243 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 29083
I20260925 02:57:59.902405 29144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:58:00.003716 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 29083
2026-09-25T02:58:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2294 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:58:00 dist-test-slave-8zvk krb5kdc[29579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:58:00 dist-test-slave-8zvk krb5kdc[29579](info): set up 2 sockets
Sep 25 02:58:00 dist-test-slave-8zvk krb5kdc[29579](info): commencing operation
krb5kdc: starting...
W20260925 02:58:01.095289 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 02:58:01 dist-test-slave-8zvk krb5kdc[29579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:58:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:58:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 02:58:01.324517 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:33605
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:33605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:58:01.653383 29592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:58:01.653620 29592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:58:01.653698 29592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:58:01.660259 29592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:58:01.660338 29592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:58:01.660442 29592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:58:01.660573 29592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:58:01.660634 29592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:58:01.669613 29592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:33605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:33605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:58:01.670778 29592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:58:01.672005 29592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:58:01.683400 29598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:58:01.683318 29600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:58:01.683586 29597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:58:01.684356 29592 server_base.cc:1034] running on GCE node
I20260925 02:58:01.685077 29592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:58:01.686321 29592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:58:01.687553 29592 hybrid_clock.cc:648] HybridClock initialized: now 1790305081687495 us; error 35 us; skew 500 ppm
I20260925 02:58:01.687885 29592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:58:01 dist-test-slave-8zvk krb5kdc[29579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305081, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:58:01.690970 29592 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 02:58:01.692464 29592 webserver.cc:460] Webserver started at http://127.18.196.254:36455/ using document root <none> and password file <none>
I20260925 02:58:01.693128 29592 fs_manager.cc:359] Metadata directory not provided
I20260925 02:58:01.693281 29592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:58:01.693606 29592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:58:01.696707 29592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "24482b6822684db8aa24083814f603a0"
format_stamp: "Formatted at 2026-09-25 02:58:01 on dist-test-slave-8zvk"
server_key: "310df9de2c1be2cd57bd315ddac47546"
server_key_iv: "735462b96aa77e8c31f3c7fb4542fbf7"
server_key_version: "encryptionkey@0"
I20260925 02:58:01.697481 29592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "24482b6822684db8aa24083814f603a0"
format_stamp: "Formatted at 2026-09-25 02:58:01 on dist-test-slave-8zvk"
server_key: "310df9de2c1be2cd57bd315ddac47546"
server_key_iv: "735462b96aa77e8c31f3c7fb4542fbf7"
server_key_version: "encryptionkey@0"
I20260925 02:58:01.702489 29592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 02:58:01.705777 29607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:58:01.707018 29592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 02:58:01.707190 29592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "24482b6822684db8aa24083814f603a0"
format_stamp: "Formatted at 2026-09-25 02:58:01 on dist-test-slave-8zvk"
server_key: "310df9de2c1be2cd57bd315ddac47546"
server_key_iv: "735462b96aa77e8c31f3c7fb4542fbf7"
server_key_version: "encryptionkey@0"
I20260925 02:58:01.707376 29592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:58:01.730197 29592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:58:01.733997 29592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:58:01.734287 29592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:58:01.752537 29592 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:33605
I20260925 02:58:01.752566 29658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:33605 every 8 connection(s)
I20260925 02:58:01.753830 29592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:58:01.757339 29659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:58:01.759636 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 29592
I20260925 02:58:01.759859 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:58:01.760676 19219 external_mini_cluster.cc:1428] Setting key 1b27d3f40631c8e77d971b77f0ee5f6c
I20260925 02:58:01.772656 29659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24482b6822684db8aa24083814f603a0: Bootstrap starting.
I20260925 02:58:01.777202 29659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24482b6822684db8aa24083814f603a0: Neither blocks nor log segments found. Creating new log.
Sep 25 02:58:01 dist-test-slave-8zvk krb5kdc[29579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:58:01.778607 29659 log.cc:824] T 00000000000000000000000000000000 P 24482b6822684db8aa24083814f603a0: Log is configured to *not* fsync() on all Append() calls
I20260925 02:58:01.782080 29659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24482b6822684db8aa24083814f603a0: No bootstrap required, opened a new log
I20260925 02:58:01.787029 29662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:01.762734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41374 (local address 127.18.196.254:33605)
0925 02:58:01.763652 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:01.763690 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:01.763749 (+ 59us) server_negotiation.cc:407] Connection header received
0925 02:58:01.766493 (+ 2744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:01.766539 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:01.767146 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:01.767560 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:58:01.769588 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:01.771510 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:01.772847 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:01.774061 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:01.774270 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:01.779868 (+ 5598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:58:01.779880 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:58:01.779901 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:58:01.779955 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:58:01.783101 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:01.783898 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:01.783904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:01.783909 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:01.784080 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:01.784484 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:01.784487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:01.784489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:01.784985 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:58:01.785198 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:01.785637 (+ 439us) server_negotiation.cc:299] Negotiation successful
0925 02:58:01.785987 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":363,"threads_started":1}
I20260925 02:58:01.791127 29659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24482b6822684db8aa24083814f603a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24482b6822684db8aa24083814f603a0" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33605 } }
I20260925 02:58:01.791491 29659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24482b6822684db8aa24083814f603a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:58:01.791612 29659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24482b6822684db8aa24083814f603a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24482b6822684db8aa24083814f603a0, State: Initialized, Role: FOLLOWER
I20260925 02:58:01.792204 29659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24482b6822684db8aa24083814f603a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24482b6822684db8aa24083814f603a0" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33605 } }
I20260925 02:58:01.792439 29659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24482b6822684db8aa24083814f603a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:58:01.792591 29659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24482b6822684db8aa24083814f603a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:58:01.792768 29659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24482b6822684db8aa24083814f603a0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:58:01.795037 29659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24482b6822684db8aa24083814f603a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24482b6822684db8aa24083814f603a0" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33605 } }
I20260925 02:58:01.795594 29659 leader_election.cc:302] T 00000000000000000000000000000000 P 24482b6822684db8aa24083814f603a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24482b6822684db8aa24083814f603a0; no voters:
I20260925 02:58:01.796361 29659 leader_election.cc:288] T 00000000000000000000000000000000 P 24482b6822684db8aa24083814f603a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:58:01.796469 29664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24482b6822684db8aa24083814f603a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:58:01.797600 29664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24482b6822684db8aa24083814f603a0 [term 1 LEADER]: Becoming Leader. State: Replica: 24482b6822684db8aa24083814f603a0, State: Running, Role: LEADER
I20260925 02:58:01.798134 29664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24482b6822684db8aa24083814f603a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24482b6822684db8aa24083814f603a0" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33605 } }
I20260925 02:58:01.798694 29659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24482b6822684db8aa24083814f603a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:58:01.801156 29666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24482b6822684db8aa24083814f603a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24482b6822684db8aa24083814f603a0. Latest consensus state: current_term: 1 leader_uuid: "24482b6822684db8aa24083814f603a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24482b6822684db8aa24083814f603a0" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33605 } } }
I20260925 02:58:01.802068 29666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24482b6822684db8aa24083814f603a0 [sys.catalog]: This master's current role is: LEADER
I20260925 02:58:01.801180 29665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24482b6822684db8aa24083814f603a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24482b6822684db8aa24083814f603a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24482b6822684db8aa24083814f603a0" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33605 } } }
I20260925 02:58:01.802253 29665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24482b6822684db8aa24083814f603a0 [sys.catalog]: This master's current role is: LEADER
I20260925 02:58:01.804836 29669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:58:01.811571 29669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:58:01.821781 29669 catalog_manager.cc:1347] Generated new cluster ID: 838f8e5955884fa5ad31695022967de4
I20260925 02:58:01.821887 29669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:58:01.841732 29669 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:58:01.843134 29669 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:58:01.850406 29669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24482b6822684db8aa24083814f603a0: Generated new TSK 0
I20260925 02:58:01.851202 29669 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
I20260925 02:58:01.906950 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33605
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:58:02.249364 29685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:58:02.249616 29685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:58:02.249801 29685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:58:02.256913 29685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:58:02.257004 29685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:58:02.257148 29685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 02:58:02.266276 29685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:58:02.267884 29685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:58:02.269179 29685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:58:02.282420 29693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:58:02.282521 29690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:58:02.282452 29691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:58:02.283277 29685 server_base.cc:1034] running on GCE node
I20260925 02:58:02.283910 29685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:58:02.284801 29685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:58:02.286029 29685 hybrid_clock.cc:648] HybridClock initialized: now 1790305082285996 us; error 56 us; skew 500 ppm
I20260925 02:58:02.286327 29685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:58:02 dist-test-slave-8zvk krb5kdc[29579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305082, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:58:02.289361 29685 init.cc:377] Logged in from keytab as kudu/127.18.196.193@KRBTEST.COM (short username kudu)
I20260925 02:58:02.290798 29685 webserver.cc:460] Webserver started at http://127.18.196.193:35727/ using document root <none> and password file <none>
I20260925 02:58:02.291549 29685 fs_manager.cc:359] Metadata directory not provided
I20260925 02:58:02.291697 29685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:58:02.291996 29685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:58:02.294783 29685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "18e0ba16bd0247ad84873c6e3f3cc6b5"
format_stamp: "Formatted at 2026-09-25 02:58:02 on dist-test-slave-8zvk"
server_key: "55fee7a1c8a4d7748ec2f3c663944b2f"
server_key_iv: "c7769f3a46347d7b05942d8e01c83a80"
server_key_version: "encryptionkey@0"
I20260925 02:58:02.295651 29685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "18e0ba16bd0247ad84873c6e3f3cc6b5"
format_stamp: "Formatted at 2026-09-25 02:58:02 on dist-test-slave-8zvk"
server_key: "55fee7a1c8a4d7748ec2f3c663944b2f"
server_key_iv: "c7769f3a46347d7b05942d8e01c83a80"
server_key_version: "encryptionkey@0"
I20260925 02:58:02.300845 29685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:58:02.304198 29700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:58:02.305546 29685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 02:58:02.305725 29685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "18e0ba16bd0247ad84873c6e3f3cc6b5"
format_stamp: "Formatted at 2026-09-25 02:58:02 on dist-test-slave-8zvk"
server_key: "55fee7a1c8a4d7748ec2f3c663944b2f"
server_key_iv: "c7769f3a46347d7b05942d8e01c83a80"
server_key_version: "encryptionkey@0"
I20260925 02:58:02.305922 29685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:58:02.320225 29685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:58:02.324051 29685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:58:02.324335 29685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:58:02.325212 29685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:58:02.326444 29685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:58:02.326551 29685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:58:02.326699 29685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:58:02.326776 29685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:58:02.359818 29685 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:33179
I20260925 02:58:02.359867 29812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:33179 every 8 connection(s)
I20260925 02:58:02.361064 29685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 02:58:02.369679 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 29685
I20260925 02:58:02.369907 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 02:58:02.370529 19219 external_mini_cluster.cc:1428] Setting key 7fd4cd8be28efd5ea4e8d9ec49be6105
Sep 25 02:58:02 dist-test-slave-8zvk krb5kdc[29579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305082, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:58:02.381063 29815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:02.363484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:41177 (local address 127.18.196.254:33605)
0925 02:58:02.363893 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:02.363898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:02.364836 (+ 938us) server_negotiation.cc:407] Connection header received
0925 02:58:02.366573 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:02.366577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:02.366644 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:02.366751 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:58:02.368817 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:02.369515 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:02.371345 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:02.371876 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:02.371998 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:02.375325 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:58:02.375333 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:58:02.375341 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:58:02.375385 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:58:02.378041 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:02.378761 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:02.378771 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:02.378773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:02.378954 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:02.379395 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:02.379400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:02.379404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:02.379688 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:58:02.379848 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:02.380510 (+ 662us) server_negotiation.cc:299] Negotiation successful
0925 02:58:02.380722 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":192,"threads_started":1}
I20260925 02:58:02.381960 29816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:02.364032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:33605 (local address 127.18.196.193:41177)
0925 02:58:02.364682 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0925 02:58:02.364724 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:58:02.366303 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:58:02.366998 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:58:02.367005 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:58:02.367755 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:58:02.368652 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:02.368671 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:02.369716 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:02.369721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:02.371164 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:02.371194 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:02.372078 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:02.372082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:02.372359 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:02.373054 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:58:02.373074 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:58:02.375113 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:58:02.378183 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:58:02.378190 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:58:02.378204 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:58:02.378624 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:58:02.379069 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:58:02.379075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:58:02.379078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:58:02.379269 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:58:02.379816 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:58:02.379824 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:58:02.380138 (+ 314us) client_negotiation.cc:769] Sending connection context
0925 02:58:02.380382 (+ 244us) client_negotiation.cc:240] Negotiation successful
0925 02:58:02.380858 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":248,"threads_started":1}
I20260925 02:58:02.383584 29813 heartbeater.cc:347] Connected to a master server at 127.18.196.254:33605
I20260925 02:58:02.383881 29813 heartbeater.cc:464] Registering TS with master...
I20260925 02:58:02.384613 29813 heartbeater.cc:511] Master 127.18.196.254:33605 requested a full tablet report, sending...
I20260925 02:58:02.386812 29624 ts_manager.cc:194] Registered new tserver with Master: 18e0ba16bd0247ad84873c6e3f3cc6b5 (127.18.196.193:33179)
I20260925 02:58:02.388906 29624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.193@KRBTEST.COM'} at 127.18.196.193:41177
WARNING: no policy specified for kudu/127.18.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
I20260925 02:58:02.423501 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33605
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:58:02.733219 29819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:58:02.733480 29819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:58:02.733577 29819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:58:02.740427 29819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:58:02.740517 29819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:58:02.740624 29819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 02:58:02.749747 29819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:58:02.750933 29819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:58:02.752213 29819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:58:02.765125 29825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:58:02.765194 29827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:58:02.765595 29824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:58:02.766515 29819 server_base.cc:1034] running on GCE node
I20260925 02:58:02.767168 29819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:58:02.768172 29819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:58:02.769390 29819 hybrid_clock.cc:648] HybridClock initialized: now 1790305082769344 us; error 40 us; skew 500 ppm
I20260925 02:58:02.769650 29819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:58:02 dist-test-slave-8zvk krb5kdc[29579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305082, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:58:02.772626 29819 init.cc:377] Logged in from keytab as kudu/127.18.196.194@KRBTEST.COM (short username kudu)
I20260925 02:58:02.773924 29819 webserver.cc:460] Webserver started at http://127.18.196.194:34669/ using document root <none> and password file <none>
I20260925 02:58:02.774585 29819 fs_manager.cc:359] Metadata directory not provided
I20260925 02:58:02.774739 29819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:58:02.775066 29819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:58:02.777990 29819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "57bc7179eac342db81f2ca5e11c8a1fa"
format_stamp: "Formatted at 2026-09-25 02:58:02 on dist-test-slave-8zvk"
server_key: "b00d1e13b1ad1a2a91152a7abdc35069"
server_key_iv: "8505e03cd109482c270893ddcdcbde40"
server_key_version: "encryptionkey@0"
I20260925 02:58:02.778764 29819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "57bc7179eac342db81f2ca5e11c8a1fa"
format_stamp: "Formatted at 2026-09-25 02:58:02 on dist-test-slave-8zvk"
server_key: "b00d1e13b1ad1a2a91152a7abdc35069"
server_key_iv: "8505e03cd109482c270893ddcdcbde40"
server_key_version: "encryptionkey@0"
I20260925 02:58:02.783624 29819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 02:58:02.786743 29834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:58:02.787976 29819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:58:02.788136 29819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "57bc7179eac342db81f2ca5e11c8a1fa"
format_stamp: "Formatted at 2026-09-25 02:58:02 on dist-test-slave-8zvk"
server_key: "b00d1e13b1ad1a2a91152a7abdc35069"
server_key_iv: "8505e03cd109482c270893ddcdcbde40"
server_key_version: "encryptionkey@0"
I20260925 02:58:02.788379 29819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:58:02.806183 29819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:58:02.810088 29819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:58:02.810415 29819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:58:02.811380 29819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:58:02.812791 29819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:58:02.812865 29819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:58:02.812952 29819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:58:02.812996 29819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:58:02.844125 29819 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:40477
I20260925 02:58:02.844147 29946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:40477 every 8 connection(s)
I20260925 02:58:02.845393 29819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 02:58:02.852777 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 29819
I20260925 02:58:02.852975 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 02:58:02.853401 19219 external_mini_cluster.cc:1428] Setting key 9a2734399b873000bb3f005097e97a43
Sep 25 02:58:02 dist-test-slave-8zvk krb5kdc[29579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305082, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:58:02.865900 29815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:02.847900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:54585 (local address 127.18.196.254:33605)
0925 02:58:02.848094 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:02.848099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:02.849210 (+ 1111us) server_negotiation.cc:407] Connection header received
0925 02:58:02.850981 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:02.850986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:02.851083 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:02.851194 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:58:02.852822 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:02.853630 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:02.855646 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:02.856049 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:02.856146 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:02.860184 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:58:02.860189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:58:02.860192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:58:02.860223 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:58:02.862832 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:02.863554 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:02.863562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:02.863566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:02.863745 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:02.864260 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:02.864264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:02.864266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:02.864534 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:58:02.864644 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:02.865494 (+ 850us) server_negotiation.cc:299] Negotiation successful
0925 02:58:02.865659 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:58:02.867365 29949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:02.848357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:33605 (local address 127.18.196.194:54585)
0925 02:58:02.849058 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0925 02:58:02.849094 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:58:02.850689 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:58:02.851422 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:58:02.851430 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:58:02.851895 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:58:02.852561 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:02.852589 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:02.853795 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:02.853800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:02.855466 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:02.855500 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:02.856296 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:02.856305 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:02.856633 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:02.857339 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:58:02.857380 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:58:02.859970 (+ 2590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:58:02.863005 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:58:02.863011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:58:02.863022 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:58:02.863421 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:58:02.863872 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:58:02.863875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:58:02.863878 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:58:02.864109 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:58:02.864664 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:58:02.864675 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:58:02.865115 (+ 440us) client_negotiation.cc:769] Sending connection context
0925 02:58:02.865407 (+ 292us) client_negotiation.cc:240] Negotiation successful
0925 02:58:02.865947 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":292,"threads_started":1}
I20260925 02:58:02.868762 29947 heartbeater.cc:347] Connected to a master server at 127.18.196.254:33605
I20260925 02:58:02.869056 29947 heartbeater.cc:464] Registering TS with master...
I20260925 02:58:02.869777 29947 heartbeater.cc:511] Master 127.18.196.254:33605 requested a full tablet report, sending...
I20260925 02:58:02.871271 29624 ts_manager.cc:194] Registered new tserver with Master: 57bc7179eac342db81f2ca5e11c8a1fa (127.18.196.194:40477)
I20260925 02:58:02.872457 29624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.194@KRBTEST.COM'} at 127.18.196.194:54585
WARNING: no policy specified for kudu/127.18.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
I20260925 02:58:02.906366 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33605
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:58:03.234545 29952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:58:03.234787 29952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:58:03.234941 29952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:58:03.241343 29952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:58:03.241467 29952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:58:03.241593 29952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 02:58:03.250500 29952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:58:03.251578 29952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:58:03.252789 29952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:58:03.265578 29957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:58:03.265544 29960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:58:03.265579 29958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:58:03.266418 29952 server_base.cc:1034] running on GCE node
I20260925 02:58:03.267051 29952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:58:03.267943 29952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:58:03.269173 29952 hybrid_clock.cc:648] HybridClock initialized: now 1790305083269140 us; error 44 us; skew 500 ppm
I20260925 02:58:03.269480 29952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:58:03 dist-test-slave-8zvk krb5kdc[29579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305083, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:58:03.278707 29952 init.cc:377] Logged in from keytab as kudu/127.18.196.195@KRBTEST.COM (short username kudu)
I20260925 02:58:03.280207 29952 webserver.cc:460] Webserver started at http://127.18.196.195:43997/ using document root <none> and password file <none>
I20260925 02:58:03.280879 29952 fs_manager.cc:359] Metadata directory not provided
I20260925 02:58:03.281033 29952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:58:03.281339 29952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:58:03.284279 29952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "9a824ca6fe724cc5a102927e79b3f5d5"
format_stamp: "Formatted at 2026-09-25 02:58:03 on dist-test-slave-8zvk"
server_key: "dd4272a635e85fdf63cfeb109d70ce08"
server_key_iv: "bdee88ee5399076069467b53cdbbdf94"
server_key_version: "encryptionkey@0"
I20260925 02:58:03.285068 29952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "9a824ca6fe724cc5a102927e79b3f5d5"
format_stamp: "Formatted at 2026-09-25 02:58:03 on dist-test-slave-8zvk"
server_key: "dd4272a635e85fdf63cfeb109d70ce08"
server_key_iv: "bdee88ee5399076069467b53cdbbdf94"
server_key_version: "encryptionkey@0"
I20260925 02:58:03.290132 29952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:58:03.293767 29967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:58:03.295217 29952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:58:03.295392 29952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "9a824ca6fe724cc5a102927e79b3f5d5"
format_stamp: "Formatted at 2026-09-25 02:58:03 on dist-test-slave-8zvk"
server_key: "dd4272a635e85fdf63cfeb109d70ce08"
server_key_iv: "bdee88ee5399076069467b53cdbbdf94"
server_key_version: "encryptionkey@0"
I20260925 02:58:03.295600 29952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:58:03.314355 29952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:58:03.317945 29952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:58:03.318212 29952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:58:03.319157 29952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:58:03.320463 29952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:58:03.320533 29952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:58:03.320683 29952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:58:03.320740 29952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:58:03.353951 29952 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:36321
I20260925 02:58:03.353987 30079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:36321 every 8 connection(s)
I20260925 02:58:03.355466 29952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 02:58:03.357991 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 29952
I20260925 02:58:03.358206 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 02:58:03.358680 19219 external_mini_cluster.cc:1428] Setting key f768588c1fc275f549e5c13ab75ae422
Sep 25 02:58:03 dist-test-slave-8zvk krb5kdc[29579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305083, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:58:03.376365 29815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:03.358012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:57275 (local address 127.18.196.254:33605)
0925 02:58:03.358268 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:03.358274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:03.359632 (+ 1358us) server_negotiation.cc:407] Connection header received
0925 02:58:03.362240 (+ 2608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:03.362247 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:03.362337 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:03.362446 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:58:03.363980 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.364741 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.366253 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.366649 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.366784 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:03.370215 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:58:03.370220 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:58:03.370223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:58:03.370257 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:58:03.373527 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:03.374277 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:03.374281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:03.374283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:03.374467 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:03.374962 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:03.374965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:03.374968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:03.375241 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:58:03.375352 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:03.375961 (+ 609us) server_negotiation.cc:299] Negotiation successful
0925 02:58:03.376126 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260925 02:58:03.377240 30082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:03.358521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:33605 (local address 127.18.196.195:57275)
0925 02:58:03.359474 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0925 02:58:03.359524 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:58:03.361919 (+ 2395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:58:03.362713 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:58:03.362722 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:58:03.363272 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:58:03.363802 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:03.363820 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.364894 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.364899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:03.366102 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:03.366114 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.366898 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.366906 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:03.367139 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:03.367754 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:58:03.367782 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:58:03.369873 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:58:03.373679 (+ 3806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:58:03.373688 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:58:03.373702 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:58:03.374149 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:58:03.374590 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:58:03.374594 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:58:03.374597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:58:03.374821 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:58:03.375347 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:58:03.375353 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:58:03.375641 (+ 288us) client_negotiation.cc:769] Sending connection context
0925 02:58:03.375862 (+ 221us) client_negotiation.cc:240] Negotiation successful
0925 02:58:03.376222 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":376,"threads_started":1}
I20260925 02:58:03.378521 30080 heartbeater.cc:347] Connected to a master server at 127.18.196.254:33605
I20260925 02:58:03.378809 30080 heartbeater.cc:464] Registering TS with master...
I20260925 02:58:03.379719 30080 heartbeater.cc:511] Master 127.18.196.254:33605 requested a full tablet report, sending...
I20260925 02:58:03.381069 29624 ts_manager.cc:194] Registered new tserver with Master: 9a824ca6fe724cc5a102927e79b3f5d5 (127.18.196.195:36321)
I20260925 02:58:03.381824 29624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.195@KRBTEST.COM'} at 127.18.196.195:57275
I20260925 02:58:03.386648 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:58:03.392127 29813 heartbeater.cc:503] Master 127.18.196.254:33605 was elected leader, sending a full tablet report...
I20260925 02:58:03.403584 29815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:03.390962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41378 (local address 127.18.196.254:33605)
0925 02:58:03.391150 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:03.391159 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:03.391939 (+ 780us) server_negotiation.cc:407] Connection header received
0925 02:58:03.392062 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:03.392067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:03.392145 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:03.392253 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:58:03.393909 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.394637 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.395887 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.396304 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.396395 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:03.398519 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:58:03.398525 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:58:03.398529 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:58:03.398561 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:58:03.401198 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:03.401802 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:03.401806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:03.401808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:03.401930 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:03.402237 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:03.402240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:03.402242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:03.402559 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:58:03.402735 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:03.403075 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 02:58:03.403242 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:58:03.407428 29624 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:58:03.414335 29624 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:58:03.438488 30092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:03.425722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:36321 (local address 127.0.0.1:40160)
0925 02:58:03.426481 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0925 02:58:03.426544 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 02:58:03.426723 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:58:03.428477 (+ 1754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:58:03.428499 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:58:03.428541 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:58:03.428955 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:03.428969 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.431707 (+ 2738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.431716 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:03.434443 (+ 2727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:03.434464 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.437180 (+ 2716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.437199 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:03.437498 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:03.437572 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 02:58:03.437702 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 02:58:03.437970 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":307,"threads_started":1}
I20260925 02:58:03.440290 30093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:03.425938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40160 (local address 127.18.196.195:36321)
0925 02:58:03.427723 (+ 1785us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:03.427733 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:03.427768 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:58:03.427892 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:03.427902 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:03.428191 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:03.429154 (+ 963us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:58:03.429436 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.431518 (+ 2082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.435243 (+ 3725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.436969 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.439530 (+ 2561us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:03.439771 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:03.439857 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 02:58:03.440016 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1623,"thread_start_us":1506,"threads_started":1}
I20260925 02:58:03.444185 30015 tablet_service.cc:1467] Processing CreateTablet for tablet 0893720e5a9b47018b2a2ee7e9514b45 (DEFAULT_TABLE table=test-table [id=359dbd6e7a02476f91f554a121a724ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:58:03.445745 30094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:03.426771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:40477 (local address 127.0.0.1:48160)
0925 02:58:03.427374 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0925 02:58:03.427398 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:58:03.427521 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:58:03.439523 (+ 12002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:58:03.439540 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:58:03.439565 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:58:03.440026 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:03.440043 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.441539 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.441544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:03.443118 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:03.443131 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.444935 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.444942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:03.445142 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:03.445174 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 02:58:03.445277 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 02:58:03.445418 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":341,"threads_started":1}
I20260925 02:58:03.445767 30015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0893720e5a9b47018b2a2ee7e9514b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:58:03.445793 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:03.427319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48160 (local address 127.18.196.194:40477)
0925 02:58:03.438301 (+ 10982us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:03.438313 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:03.438349 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:58:03.438469 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:03.438480 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:03.439323 (+ 843us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:03.439528 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:58:03.440185 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.441359 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.443277 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.444733 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.444968 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:03.445156 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:03.445336 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 02:58:03.445502 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10764,"thread_start_us":11552,"threads_started":1}
I20260925 02:58:03.449086 29882 tablet_service.cc:1467] Processing CreateTablet for tablet 0893720e5a9b47018b2a2ee7e9514b45 (DEFAULT_TABLE table=test-table [id=359dbd6e7a02476f91f554a121a724ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:58:03.450140 29882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0893720e5a9b47018b2a2ee7e9514b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:58:03.450788 30092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:03.427445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33179 (local address 127.0.0.1:40196)
0925 02:58:03.439098 (+ 11653us) negotiation.cc:107] Waiting for socket to connect
0925 02:58:03.439122 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:58:03.439238 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:58:03.444347 (+ 5109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:58:03.444352 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:58:03.444380 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:58:03.444565 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:03.444577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.446539 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.446545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:03.448314 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:03.448329 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.450126 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.450131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:03.450298 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:03.450325 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 02:58:03.450390 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 02:58:03.450526 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11528,"thread_start_us":12437,"threads_started":1}
I20260925 02:58:03.451874 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:03.428007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40196 (local address 127.18.196.193:33179)
0925 02:58:03.443795 (+ 15788us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:03.443802 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:03.443829 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:58:03.443920 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:03.443926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:03.444138 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:03.444334 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:58:03.444795 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.446392 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.448490 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.449939 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.451176 (+ 1237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:03.451340 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:03.451406 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 02:58:03.451560 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15609,"thread_start_us":15413,"threads_started":1}
I20260925 02:58:03.455317 29748 tablet_service.cc:1467] Processing CreateTablet for tablet 0893720e5a9b47018b2a2ee7e9514b45 (DEFAULT_TABLE table=test-table [id=359dbd6e7a02476f91f554a121a724ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:58:03.456620 29748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0893720e5a9b47018b2a2ee7e9514b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:58:03.459509 30098 tablet_bootstrap.cc:492] T 0893720e5a9b47018b2a2ee7e9514b45 P 9a824ca6fe724cc5a102927e79b3f5d5: Bootstrap starting.
I20260925 02:58:03.462052 30099 tablet_bootstrap.cc:492] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa: Bootstrap starting.
I20260925 02:58:03.463690 30098 tablet_bootstrap.cc:654] T 0893720e5a9b47018b2a2ee7e9514b45 P 9a824ca6fe724cc5a102927e79b3f5d5: Neither blocks nor log segments found. Creating new log.
I20260925 02:58:03.464658 30098 log.cc:824] T 0893720e5a9b47018b2a2ee7e9514b45 P 9a824ca6fe724cc5a102927e79b3f5d5: Log is configured to *not* fsync() on all Append() calls
I20260925 02:58:03.465265 30099 tablet_bootstrap.cc:654] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa: Neither blocks nor log segments found. Creating new log.
I20260925 02:58:03.466432 30099 log.cc:824] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa: Log is configured to *not* fsync() on all Append() calls
I20260925 02:58:03.467362 30098 tablet_bootstrap.cc:492] T 0893720e5a9b47018b2a2ee7e9514b45 P 9a824ca6fe724cc5a102927e79b3f5d5: No bootstrap required, opened a new log
I20260925 02:58:03.467653 30098 ts_tablet_manager.cc:1397] T 0893720e5a9b47018b2a2ee7e9514b45 P 9a824ca6fe724cc5a102927e79b3f5d5: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 02:58:03.469323 30099 tablet_bootstrap.cc:492] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa: No bootstrap required, opened a new log
I20260925 02:58:03.469586 30099 ts_tablet_manager.cc:1397] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 02:58:03.470709 30102 tablet_bootstrap.cc:492] T 0893720e5a9b47018b2a2ee7e9514b45 P 18e0ba16bd0247ad84873c6e3f3cc6b5: Bootstrap starting.
I20260925 02:58:03.474155 30102 tablet_bootstrap.cc:654] T 0893720e5a9b47018b2a2ee7e9514b45 P 18e0ba16bd0247ad84873c6e3f3cc6b5: Neither blocks nor log segments found. Creating new log.
I20260925 02:58:03.475488 30102 log.cc:824] T 0893720e5a9b47018b2a2ee7e9514b45 P 18e0ba16bd0247ad84873c6e3f3cc6b5: Log is configured to *not* fsync() on all Append() calls
I20260925 02:58:03.477412 30098 raft_consensus.cc:348] T 0893720e5a9b47018b2a2ee7e9514b45 P 9a824ca6fe724cc5a102927e79b3f5d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a824ca6fe724cc5a102927e79b3f5d5" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 36321 } } peers { permanent_uuid: "57bc7179eac342db81f2ca5e11c8a1fa" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40477 } } peers { permanent_uuid: "18e0ba16bd0247ad84873c6e3f3cc6b5" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33179 } }
I20260925 02:58:03.477550 30099 raft_consensus.cc:348] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a824ca6fe724cc5a102927e79b3f5d5" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 36321 } } peers { permanent_uuid: "57bc7179eac342db81f2ca5e11c8a1fa" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40477 } } peers { permanent_uuid: "18e0ba16bd0247ad84873c6e3f3cc6b5" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33179 } }
I20260925 02:58:03.477876 30098 raft_consensus.cc:374] T 0893720e5a9b47018b2a2ee7e9514b45 P 9a824ca6fe724cc5a102927e79b3f5d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:58:03.477974 30098 raft_consensus.cc:729] T 0893720e5a9b47018b2a2ee7e9514b45 P 9a824ca6fe724cc5a102927e79b3f5d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a824ca6fe724cc5a102927e79b3f5d5, State: Initialized, Role: FOLLOWER
I20260925 02:58:03.478005 30099 raft_consensus.cc:374] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:58:03.478106 30099 raft_consensus.cc:729] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57bc7179eac342db81f2ca5e11c8a1fa, State: Initialized, Role: FOLLOWER
I20260925 02:58:03.478685 30102 tablet_bootstrap.cc:492] T 0893720e5a9b47018b2a2ee7e9514b45 P 18e0ba16bd0247ad84873c6e3f3cc6b5: No bootstrap required, opened a new log
I20260925 02:58:03.478674 30099 consensus_queue.cc:260] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a824ca6fe724cc5a102927e79b3f5d5" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 36321 } } peers { permanent_uuid: "57bc7179eac342db81f2ca5e11c8a1fa" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40477 } } peers { permanent_uuid: "18e0ba16bd0247ad84873c6e3f3cc6b5" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33179 } }
I20260925 02:58:03.478744 30098 consensus_queue.cc:260] T 0893720e5a9b47018b2a2ee7e9514b45 P 9a824ca6fe724cc5a102927e79b3f5d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a824ca6fe724cc5a102927e79b3f5d5" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 36321 } } peers { permanent_uuid: "57bc7179eac342db81f2ca5e11c8a1fa" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40477 } } peers { permanent_uuid: "18e0ba16bd0247ad84873c6e3f3cc6b5" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33179 } }
I20260925 02:58:03.478991 30102 ts_tablet_manager.cc:1397] T 0893720e5a9b47018b2a2ee7e9514b45 P 18e0ba16bd0247ad84873c6e3f3cc6b5: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260925 02:58:03.479794 29947 heartbeater.cc:503] Master 127.18.196.254:33605 was elected leader, sending a full tablet report...
I20260925 02:58:03.481838 30099 ts_tablet_manager.cc:1428] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa: Time spent starting tablet: real 0.012s user 0.005s sys 0.007s
I20260925 02:58:03.482760 30080 heartbeater.cc:503] Master 127.18.196.254:33605 was elected leader, sending a full tablet report...
I20260925 02:58:03.483373 30098 ts_tablet_manager.cc:1428] T 0893720e5a9b47018b2a2ee7e9514b45 P 9a824ca6fe724cc5a102927e79b3f5d5: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260925 02:58:03.489396 30102 raft_consensus.cc:348] T 0893720e5a9b47018b2a2ee7e9514b45 P 18e0ba16bd0247ad84873c6e3f3cc6b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a824ca6fe724cc5a102927e79b3f5d5" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 36321 } } peers { permanent_uuid: "57bc7179eac342db81f2ca5e11c8a1fa" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40477 } } peers { permanent_uuid: "18e0ba16bd0247ad84873c6e3f3cc6b5" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33179 } }
I20260925 02:58:03.489746 30102 raft_consensus.cc:374] T 0893720e5a9b47018b2a2ee7e9514b45 P 18e0ba16bd0247ad84873c6e3f3cc6b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:58:03.489866 30102 raft_consensus.cc:729] T 0893720e5a9b47018b2a2ee7e9514b45 P 18e0ba16bd0247ad84873c6e3f3cc6b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18e0ba16bd0247ad84873c6e3f3cc6b5, State: Initialized, Role: FOLLOWER
I20260925 02:58:03.490397 30102 consensus_queue.cc:260] T 0893720e5a9b47018b2a2ee7e9514b45 P 18e0ba16bd0247ad84873c6e3f3cc6b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a824ca6fe724cc5a102927e79b3f5d5" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 36321 } } peers { permanent_uuid: "57bc7179eac342db81f2ca5e11c8a1fa" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40477 } } peers { permanent_uuid: "18e0ba16bd0247ad84873c6e3f3cc6b5" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33179 } }
I20260925 02:58:03.491881 30102 ts_tablet_manager.cc:1428] T 0893720e5a9b47018b2a2ee7e9514b45 P 18e0ba16bd0247ad84873c6e3f3cc6b5: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
I20260925 02:58:03.511282 30104 raft_consensus.cc:482] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:58:03.511538 30104 raft_consensus.cc:504] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a824ca6fe724cc5a102927e79b3f5d5" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 36321 } } peers { permanent_uuid: "57bc7179eac342db81f2ca5e11c8a1fa" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40477 } } peers { permanent_uuid: "18e0ba16bd0247ad84873c6e3f3cc6b5" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33179 } }
I20260925 02:58:03.513193 30104 leader_election.cc:288] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a824ca6fe724cc5a102927e79b3f5d5 (127.18.196.195:36321), 18e0ba16bd0247ad84873c6e3f3cc6b5 (127.18.196.193:33179)
I20260925 02:58:03.519909 30093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:03.513572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:40347 (local address 127.18.196.195:36321)
0925 02:58:03.513825 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:03.513833 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:03.514553 (+ 720us) server_negotiation.cc:407] Connection header received
0925 02:58:03.514764 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:03.514771 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:03.514874 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:03.514963 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:58:03.515550 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.516311 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.518144 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.518967 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.519152 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:03.519225 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:03.519536 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 02:58:03.519654 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 02:58:03.519984 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:03.513612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:48789 (local address 127.18.196.193:33179)
0925 02:58:03.513825 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:03.513833 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:03.514648 (+ 815us) server_negotiation.cc:407] Connection header received
0925 02:58:03.514898 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:03.514902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:03.514960 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:03.515071 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:58:03.515704 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.516404 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.518199 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.519051 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.519231 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:03.519304 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:03.519608 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 02:58:03.519737 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:58:03.520051 30108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:03.513382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:36321 (local address 127.18.196.194:40347)
0925 02:58:03.514443 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0925 02:58:03.514459 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:58:03.514627 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:58:03.514999 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:58:03.515003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:58:03.515029 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:58:03.515397 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:03.515410 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.516432 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.516440 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:03.517976 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:03.517999 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.519177 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.519183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:03.519365 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:03.519388 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:58:03.519501 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 02:58:03.519620 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"mutex_wait_us":1,"thread_start_us":756,"threads_started":1}
I20260925 02:58:03.520437 30107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:03.513607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33179 (local address 127.18.196.194:48789)
0925 02:58:03.514544 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0925 02:58:03.514558 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:58:03.514715 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:58:03.515090 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:58:03.515094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:58:03.515132 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:58:03.515539 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:03.515548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.516561 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.516565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:03.518040 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:03.518053 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.519262 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.519269 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:03.519441 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:03.519463 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:58:03.519583 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 02:58:03.519700 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":3,"thread_start_us":456,"threads_started":1}
I20260925 02:58:03.521157 30035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0893720e5a9b47018b2a2ee7e9514b45" candidate_uuid: "57bc7179eac342db81f2ca5e11c8a1fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a824ca6fe724cc5a102927e79b3f5d5" is_pre_election: true
I20260925 02:58:03.521389 29768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0893720e5a9b47018b2a2ee7e9514b45" candidate_uuid: "57bc7179eac342db81f2ca5e11c8a1fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18e0ba16bd0247ad84873c6e3f3cc6b5" is_pre_election: true
I20260925 02:58:03.521739 30035 raft_consensus.cc:2454] T 0893720e5a9b47018b2a2ee7e9514b45 P 9a824ca6fe724cc5a102927e79b3f5d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57bc7179eac342db81f2ca5e11c8a1fa in term 0.
I20260925 02:58:03.521987 29768 raft_consensus.cc:2454] T 0893720e5a9b47018b2a2ee7e9514b45 P 18e0ba16bd0247ad84873c6e3f3cc6b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57bc7179eac342db81f2ca5e11c8a1fa in term 0.
I20260925 02:58:03.522553 29836 leader_election.cc:302] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57bc7179eac342db81f2ca5e11c8a1fa, 9a824ca6fe724cc5a102927e79b3f5d5; no voters:
I20260925 02:58:03.523008 30104 raft_consensus.cc:2781] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:58:03.523157 30104 raft_consensus.cc:482] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:58:03.523269 30104 raft_consensus.cc:3037] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:58:03.525352 30104 raft_consensus.cc:504] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a824ca6fe724cc5a102927e79b3f5d5" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 36321 } } peers { permanent_uuid: "57bc7179eac342db81f2ca5e11c8a1fa" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40477 } } peers { permanent_uuid: "18e0ba16bd0247ad84873c6e3f3cc6b5" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33179 } }
I20260925 02:58:03.525935 30104 leader_election.cc:288] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa [CANDIDATE]: Term 1 election: Requested vote from peers 9a824ca6fe724cc5a102927e79b3f5d5 (127.18.196.195:36321), 18e0ba16bd0247ad84873c6e3f3cc6b5 (127.18.196.193:33179)
I20260925 02:58:03.526258 30035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0893720e5a9b47018b2a2ee7e9514b45" candidate_uuid: "57bc7179eac342db81f2ca5e11c8a1fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a824ca6fe724cc5a102927e79b3f5d5"
I20260925 02:58:03.526440 30035 raft_consensus.cc:3037] T 0893720e5a9b47018b2a2ee7e9514b45 P 9a824ca6fe724cc5a102927e79b3f5d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:58:03.526393 29768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0893720e5a9b47018b2a2ee7e9514b45" candidate_uuid: "57bc7179eac342db81f2ca5e11c8a1fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18e0ba16bd0247ad84873c6e3f3cc6b5"
I20260925 02:58:03.526620 29768 raft_consensus.cc:3037] T 0893720e5a9b47018b2a2ee7e9514b45 P 18e0ba16bd0247ad84873c6e3f3cc6b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:58:03.528697 30035 raft_consensus.cc:2454] T 0893720e5a9b47018b2a2ee7e9514b45 P 9a824ca6fe724cc5a102927e79b3f5d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57bc7179eac342db81f2ca5e11c8a1fa in term 1.
I20260925 02:58:03.528872 29768 raft_consensus.cc:2454] T 0893720e5a9b47018b2a2ee7e9514b45 P 18e0ba16bd0247ad84873c6e3f3cc6b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57bc7179eac342db81f2ca5e11c8a1fa in term 1.
I20260925 02:58:03.529057 29836 leader_election.cc:302] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57bc7179eac342db81f2ca5e11c8a1fa, 9a824ca6fe724cc5a102927e79b3f5d5; no voters:
I20260925 02:58:03.529343 30104 raft_consensus.cc:2781] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:58:03.529772 30104 raft_consensus.cc:686] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa [term 1 LEADER]: Becoming Leader. State: Replica: 57bc7179eac342db81f2ca5e11c8a1fa, State: Running, Role: LEADER
I20260925 02:58:03.530270 30104 consensus_queue.cc:237] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a824ca6fe724cc5a102927e79b3f5d5" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 36321 } } peers { permanent_uuid: "57bc7179eac342db81f2ca5e11c8a1fa" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40477 } } peers { permanent_uuid: "18e0ba16bd0247ad84873c6e3f3cc6b5" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33179 } }
I20260925 02:58:03.534402 29624 catalog_manager.cc:5495] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 57bc7179eac342db81f2ca5e11c8a1fa (127.18.196.194). New cstate: current_term: 1 leader_uuid: "57bc7179eac342db81f2ca5e11c8a1fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a824ca6fe724cc5a102927e79b3f5d5" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57bc7179eac342db81f2ca5e11c8a1fa" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18e0ba16bd0247ad84873c6e3f3cc6b5" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33179 } health_report { overall_health: UNKNOWN } } }
I20260925 02:58:03.557528 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:03.552547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48168 (local address 127.18.196.194:40477)
0925 02:58:03.552772 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:03.552778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:03.552795 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:58:03.552979 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:03.552984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:03.553054 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:03.553201 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:58:03.553558 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.554231 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.555555 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.555945 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.556073 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:03.556519 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:58:03.556957 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:58:03.557107 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:03.557193 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 02:58:03.557268 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 02:58:03.565140 30035 raft_consensus.cc:1261] T 0893720e5a9b47018b2a2ee7e9514b45 P 9a824ca6fe724cc5a102927e79b3f5d5 [term 1 FOLLOWER]: Refusing update from remote peer 57bc7179eac342db81f2ca5e11c8a1fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:58:03.565141 29768 raft_consensus.cc:1261] T 0893720e5a9b47018b2a2ee7e9514b45 P 18e0ba16bd0247ad84873c6e3f3cc6b5 [term 1 FOLLOWER]: Refusing update from remote peer 57bc7179eac342db81f2ca5e11c8a1fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:58:03.566043 30109 consensus_queue.cc:1035] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "18e0ba16bd0247ad84873c6e3f3cc6b5" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:58:03.566347 30104 consensus_queue.cc:1035] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a824ca6fe724cc5a102927e79b3f5d5" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:58:03.590472 29623 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:58:03.590799 29623 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:58:03.594662 29623 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24482b6822684db8aa24083814f603a0: Sending DeleteTablet for 3 replicas of tablet 0893720e5a9b47018b2a2ee7e9514b45
I20260925 02:58:03.595840 29882 tablet_service.cc:1514] Processing DeleteTablet for tablet 0893720e5a9b47018b2a2ee7e9514b45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:58:03 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:48160
I20260925 02:58:03.595824 30015 tablet_service.cc:1514] Processing DeleteTablet for tablet 0893720e5a9b47018b2a2ee7e9514b45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:58:03 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:40160
I20260925 02:58:03.595870 29748 tablet_service.cc:1514] Processing DeleteTablet for tablet 0893720e5a9b47018b2a2ee7e9514b45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:58:03 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:40196
I20260925 02:58:03.596755 30122 tablet_replica.cc:330] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa: stopping tablet replica
I20260925 02:58:03.597227 30122 raft_consensus.cc:2229] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:58:03.597831 30124 tablet_replica.cc:330] T 0893720e5a9b47018b2a2ee7e9514b45 P 9a824ca6fe724cc5a102927e79b3f5d5: stopping tablet replica
I20260925 02:58:03.598044 30122 raft_consensus.cc:2258] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:58:03.598233 30124 raft_consensus.cc:2229] T 0893720e5a9b47018b2a2ee7e9514b45 P 9a824ca6fe724cc5a102927e79b3f5d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:58:03.598431 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 29685
I20260925 02:58:03.598771 30124 raft_consensus.cc:2258] T 0893720e5a9b47018b2a2ee7e9514b45 P 9a824ca6fe724cc5a102927e79b3f5d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:58:03.599807 30123 tablet_replica.cc:330] T 0893720e5a9b47018b2a2ee7e9514b45 P 18e0ba16bd0247ad84873c6e3f3cc6b5: stopping tablet replica
I20260925 02:58:03.600215 30123 raft_consensus.cc:2229] T 0893720e5a9b47018b2a2ee7e9514b45 P 18e0ba16bd0247ad84873c6e3f3cc6b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:58:03.600684 30123 raft_consensus.cc:2258] T 0893720e5a9b47018b2a2ee7e9514b45 P 18e0ba16bd0247ad84873c6e3f3cc6b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:58:03.601732 30122 ts_tablet_manager.cc:1905] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:58:03.602404 30124 ts_tablet_manager.cc:1905] T 0893720e5a9b47018b2a2ee7e9514b45 P 9a824ca6fe724cc5a102927e79b3f5d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:58:03.608987 30124 ts_tablet_manager.cc:1918] T 0893720e5a9b47018b2a2ee7e9514b45 P 9a824ca6fe724cc5a102927e79b3f5d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:58:03.609161 30124 log.cc:1196] T 0893720e5a9b47018b2a2ee7e9514b45 P 9a824ca6fe724cc5a102927e79b3f5d5: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/0893720e5a9b47018b2a2ee7e9514b45
I20260925 02:58:03.610409 30122 ts_tablet_manager.cc:1918] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:58:03.610733 30122 log.cc:1196] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/0893720e5a9b47018b2a2ee7e9514b45
I20260925 02:58:03.611521 30122 ts_tablet_manager.cc:1939] T 0893720e5a9b47018b2a2ee7e9514b45 P 57bc7179eac342db81f2ca5e11c8a1fa: Deleting consensus metadata
I20260925 02:58:03.611716 30123 ts_tablet_manager.cc:1905] T 0893720e5a9b47018b2a2ee7e9514b45 P 18e0ba16bd0247ad84873c6e3f3cc6b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:58:03.613409 29609 catalog_manager.cc:4841] TS 57bc7179eac342db81f2ca5e11c8a1fa (127.18.196.194:40477): tablet 0893720e5a9b47018b2a2ee7e9514b45 (table test-table [id=359dbd6e7a02476f91f554a121a724ed]) successfully deleted
Sep 25 02:58:03 dist-test-slave-8zvk krb5kdc[29579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.193@KRBTEST.COM
I20260925 02:58:03.617447 30123 ts_tablet_manager.cc:1918] T 0893720e5a9b47018b2a2ee7e9514b45 P 18e0ba16bd0247ad84873c6e3f3cc6b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:58:03.617605 30123 log.cc:1196] T 0893720e5a9b47018b2a2ee7e9514b45 P 18e0ba16bd0247ad84873c6e3f3cc6b5: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304987644417-19219-0/minicluster-data/ts-0/wal/wals/0893720e5a9b47018b2a2ee7e9514b45
I20260925 02:58:03.618152 30123 ts_tablet_manager.cc:1939] T 0893720e5a9b47018b2a2ee7e9514b45 P 18e0ba16bd0247ad84873c6e3f3cc6b5: Deleting consensus metadata
I20260925 02:58:03.609665 30124 ts_tablet_manager.cc:1939] T 0893720e5a9b47018b2a2ee7e9514b45 P 9a824ca6fe724cc5a102927e79b3f5d5: Deleting consensus metadata
I20260925 02:58:03.619931 29610 catalog_manager.cc:4841] TS 18e0ba16bd0247ad84873c6e3f3cc6b5 (127.18.196.193:33179): tablet 0893720e5a9b47018b2a2ee7e9514b45 (table test-table [id=359dbd6e7a02476f91f554a121a724ed]) successfully deleted
I20260925 02:58:03.620874 29609 catalog_manager.cc:4841] TS 9a824ca6fe724cc5a102927e79b3f5d5 (127.18.196.195:36321): tablet 0893720e5a9b47018b2a2ee7e9514b45 (table test-table [id=359dbd6e7a02476f91f554a121a724ed]) successfully deleted
I20260925 02:58:03.625303 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:03.601837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40212 (local address 127.18.196.193:33179)
0925 02:58:03.602017 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:03.602023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:03.606678 (+ 4655us) server_negotiation.cc:407] Connection header received
0925 02:58:03.606971 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:03.606977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:03.607070 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:03.607174 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:58:03.609070 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.610225 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.612351 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.613139 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.613239 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:03.618546 (+ 5307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:58:03.618560 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:58:03.618578 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:58:03.618618 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:58:03.622551 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:03.623296 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:03.623304 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:03.623307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:03.623477 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:03.623829 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:03.623834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:03.623837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:03.624285 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:58:03.624582 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:03.624735 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 02:58:03.624944 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:58:03.625913 29808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:58:03.752252 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 29685
I20260925 02:58:03.775656 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 29819
Sep 25 02:58:03 dist-test-slave-8zvk krb5kdc[29579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.194@KRBTEST.COM
I20260925 02:58:03.793815 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:03.776584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48184 (local address 127.18.196.194:40477)
0925 02:58:03.776778 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:03.776784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:03.776804 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:58:03.776954 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:03.776960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:03.777068 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:03.777177 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:58:03.778720 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.780133 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.781935 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.782615 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.782749 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:03.787758 (+ 5009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:58:03.787770 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:58:03.787785 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:58:03.787825 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:58:03.790690 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:03.791597 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:03.791605 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:03.791608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:03.791805 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:03.792247 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:03.792251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:03.792253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:03.792618 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:58:03.792861 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:03.793277 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 02:58:03.793439 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 02:58:03.794584 29942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:58:03.908077 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 29819
I20260925 02:58:03.926978 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 29952
Sep 25 02:58:03 dist-test-slave-8zvk krb5kdc[29579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.195@KRBTEST.COM
I20260925 02:58:03.942176 30093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:03.927705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.18.196.195:36321)
0925 02:58:03.927883 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:03.927889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:03.927949 (+ 60us) server_negotiation.cc:407] Connection header received
0925 02:58:03.928163 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:03.928167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:03.928236 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:03.928338 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:58:03.929611 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.930359 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.931846 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:03.932535 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:03.932640 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:03.937141 (+ 4501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:58:03.937153 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:58:03.937166 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:58:03.937195 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:58:03.939888 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:03.940422 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:03.940428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:03.940431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:03.940604 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:03.940887 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:03.940889 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:03.940891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:03.941211 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:58:03.941443 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:03.941658 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 02:58:03.941810 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:58:03.942912 30075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:58:04.028446 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 29952
I20260925 02:58:04.046087 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 29592
I20260925 02:58:04.047279 29654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:58:04.152352 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 29592
2026-09-25T02:58:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4149 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:58:04 dist-test-slave-8zvk krb5kdc[30132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:58:04 dist-test-slave-8zvk krb5kdc[30132](info): set up 2 sockets
Sep 25 02:58:04 dist-test-slave-8zvk krb5kdc[30132](info): commencing operation
krb5kdc: starting...
W20260925 02:58:05.243140 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 02:58:05 dist-test-slave-8zvk krb5kdc[30132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:58:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:58:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.18.196.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.18.196.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.18.196.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/rangeradmin_127.18.196.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.18.196.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/rangeradmin_127.18.196.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.18.196.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.18.196.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.18.196.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/rangerlookup_127.18.196.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.18.196.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/rangerlookup_127.18.196.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.196.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.196.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.196.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/HTTP_127.18.196.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.196.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/HTTP_127.18.196.212@KRBTEST.COM.keytab.
I20260925 02:58:05.546134 19219 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskM3GXH0/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/postgres -l logfile start
2026-09-25 02:58:08.066 UTC [30162] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 02:58:08.066 UTC [30162] LOG: listening on IPv4 address "127.18.196.212", port 46923
2026-09-25 02:58:08.070 UTC [30162] LOG: listening on Unix socket "/tmp/.s.PGSQL.46923"
2026-09-25 02:58:08.092 UTC [30165] LOG: database system was shut down at 2026-09-25 02:58:06 UTC
2026-09-25 02:58:08.096 UTC [30162] LOG: database system is ready to accept connections
I20260925 02:58:09.062780 19219 mini_postgres.cc:96] Postgres bound to 46923
2026-09-25 02:58:09.081 UTC [30173] FATAL: database "slave" does not exist
127.18.196.212:46923 - accepting connections
I20260925 02:58:09.082577 19219 mini_ranger.cc:161] Starting Ranger...
I20260925 02:58:09.108963 19219 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 02:58:09.563094 19219 mini_ranger.cc:87] Created ranger Postgres database
I20260925 02:58:09.563239 19219 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ranger-admin
2026-09-25 02:58:10,037 [I] DB FLAVOR :POSTGRES
2026-09-25 02:58:10,038 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:58:10,038 [I] Checking connection..
2026-09-25 02:58:10,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:46923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 02:58:10,302 [I] Checking connection passed.
2026-09-25 02:58:10,302 [I] --------- Verifying version history table ---------
2026-09-25 02:58:10,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:46923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:58:10,555 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:58:10,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:46923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:58:10,781 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:58:10,781 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 02:58:10,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:46923/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 02:58:11.009 UTC [30237] WARNING: there is no transaction in progress
2026-09-25 02:58:11,015 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 02:58:11,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:46923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:58:11,245 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 02:58:11,245 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:58:11,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:46923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 02:58:11,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:46923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 02:58:11,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:46923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8zvk.c.gcp-upstream.internal','N') ;"
2026-09-25 02:58:11,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:46923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:58:12,240 [I] Table x_portal_user does not exist in database ranger
2026-09-25 02:58:12,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:46923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:58:12,461 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 02:58:12,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:46923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:58:12,701 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:46923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:58:12,922 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 02:58:12,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:46923/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 02:58:13.866 UTC [30345] WARNING: there is no transaction in progress
2026-09-25 02:58:13.882 UTC [30345] WARNING: there is no transaction in progress
2026-09-25 02:58:13.898 UTC [30345] WARNING: there is no transaction in progress
2026-09-25 02:58:13.913 UTC [30345] WARNING: there is no transaction in progress
2026-09-25 02:58:13.929 UTC [30345] WARNING: there is no transaction in progress
2026-09-25 02:58:13.946 UTC [30345] WARNING: there is no transaction in progress
2026-09-25 02:58:14.004 UTC [30345] WARNING: there is no transaction in progress
2026-09-25 02:58:14.011 UTC [30345] WARNING: there is no transaction in progress
2026-09-25 02:58:14.024 UTC [30345] WARNING: there is no transaction in progress
2026-09-25 02:58:14.034 UTC [30345] WARNING: there is no transaction in progress
2026-09-25 02:58:14.043 UTC [30345] WARNING: there is no transaction in progress
2026-09-25 02:58:14.056 UTC [30345] WARNING: there is no transaction in progress
2026-09-25 02:58:14.065 UTC [30345] WARNING: there is no transaction in progress
2026-09-25 02:58:14.071 UTC [30345] WARNING: there is no transaction in progress
2026-09-25 02:58:14.077 UTC [30345] WARNING: there is no transaction in progress
2026-09-25 02:58:14.084 UTC [30345] WARNING: there is no transaction in progress
2026-09-25 02:58:14,695 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 02:58:14,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:46923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 02:58:14,907 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 02:58:14,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:46923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:58:15,161 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 02:58:15,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:46923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:58:15,411 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 02:58:15,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:46923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:58:15,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:46923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:58:15,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:46923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8zvk.c.gcp-upstream.internal';"
2026-09-25 02:58:16,109 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 02:58:16,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:46923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 02:58:16,337 [I] DB_PATCHES have already been applied
I20260925 02:58:16.343374 19219 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ranger-admin:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskM3GXH0/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 02:58:16.343446 19219 mini_ranger.cc:193] Using host: 127.18.196.212
I20260925 02:58:16.358026 19219 mini_ranger.cc:227] Ranger admin URL: http://127.18.196.212:55313
Sep 25, 2026 2:58:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:58:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:58:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:58:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:58:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:58:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:58:17 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 02:58:17.465235 19219 mini_ranger.cc:160] Time spent starting Ranger: real 8.383s user 0.007s sys 0.031s
Sep 25, 2026 2:58:38 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:58:38 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 2:58:38 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:58:38 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:58:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:58:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:58:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:58:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:58:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:58:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:58:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:58:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:58:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:58:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:58:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:58:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:58:38 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:58:39 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 02:58:39.597744 19219 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.18.196.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.18.196.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.18.196.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/rangerkms_127.18.196.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.18.196.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/rangerkms_127.18.196.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.196.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.18.196.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.196.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/HTTP_127.18.196.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.196.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/HTTP_127.18.196.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 02:58:39 dist-test-slave-8zvk krb5kdc[30132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305119, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 02:58:39.762233 19219 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 02:58:39.778710 19219 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 02:58:40.644706 19219 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 02:58:40.644870 19219 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ranger-kms
I20260925 02:58:40.645010 19219 mini_ranger_kms.cc:227] Using postgres at 127.18.196.212:46923
2026-09-25 02:58:41,489 [I] DB FLAVOR :POSTGRES
2026-09-25 02:58:41,489 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:58:41,489 [I] Checking connection
2026-09-25 02:58:41,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:46923/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 02:58:41,741 [I] connection success
2026-09-25 02:58:41,742 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 02:58:41,742 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 02:58:41,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:46923/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 02:58:41,968 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 02:58:41,968 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:58:41,968 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 02:58:41,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:46923/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 02:58:42,265 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 02:58:42.741278 19219 mini_ranger_kms.cc:310] Created kms service
I20260925 02:58:42.935169 19219 mini_ranger_kms.cc:326] Created kudu user
I20260925 02:58:42.982194 19219 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 02:58:42 dist-test-slave-8zvk krb5kdc[30132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305119, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.18.196.212@KRBTEST.COM
I20260925 02:58:43.375314 19219 mini_ranger_kms.cc:384] Added ranger policy
I20260925 02:58:43.376469 19219 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ranger-kms:/tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 02:58:43.376555 19219 mini_ranger_kms.cc:240] Using host: 127.18.196.212
I20260925 02:58:43.396549 19219 mini_ranger_kms.cc:276] Ranger KMS PID: 30526
I20260925 02:58:43.396778 19219 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.18.196.212:51559
Sep 25, 2026 2:58:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:58:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:58:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:58:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:58:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:58:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 02:58:45.755573 19219 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.993s user 0.125s sys 0.407s
I20260925 02:58:45.755923 19219 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 02:58:45.756129 19219 mini_ranger_kms.cc:401] 127.18.196.212:51559/kms/v1/keys
Sep 25 02:58:45 dist-test-slave-8zvk krb5kdc[30132](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790305125, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.196.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 02:58:46 dist-test-slave-8zvk krb5kdc[30132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305125, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.196.212@KRBTEST.COM for HTTP/127.18.196.212@KRBTEST.COM
Sep 25 02:58:46 dist-test-slave-8zvk krb5kdc[30132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305125, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.196.212@KRBTEST.COM for HTTP/127.18.196.212@KRBTEST.COM
Sep 25 02:58:48 dist-test-slave-8zvk krb5kdc[30132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 02:58:49.008759 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:37027
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:38657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:37027
--ranger_config_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.196.212:51559/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:58:49.408747 30578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:58:49.409075 30578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:58:49.409267 30578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:58:49.416931 30578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:58:49.417093 30578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:58:49.417205 30578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:58:49.417305 30578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:58:49.417392 30578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:58:49.428391 30578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:38657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/master-0/wal
--ranger_kms_url=127.18.196.212:51559/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:37027
--ranger_config_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:37027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:58:49.429668 30578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:58:49.431039 30578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:58:49.443550 30584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:58:49.443653 30586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:58:49.443749 30583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:58:49.445108 30578 server_base.cc:1034] running on GCE node
I20260925 02:58:49.445853 30578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:58:49.447242 30578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:58:49.448523 30578 hybrid_clock.cc:648] HybridClock initialized: now 1790305129448434 us; error 28 us; skew 500 ppm
I20260925 02:58:49.448989 30578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:58:49 dist-test-slave-8zvk krb5kdc[30132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305129, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:58:49.455417 30578 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 02:58:49.456888 30578 webserver.cc:460] Webserver started at http://127.18.196.254:45861/ using document root <none> and password file <none>
I20260925 02:58:49.457511 30578 fs_manager.cc:359] Metadata directory not provided
I20260925 02:58:49.457646 30578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:58:49.457947 30578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:58:49 dist-test-slave-8zvk krb5kdc[30132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305129, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for HTTP/127.18.196.212@KRBTEST.COM
I20260925 02:58:49.518106 30578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "0d3cf55d13334ed0b020b905e862f2af"
format_stamp: "Formatted at 2026-09-25 02:58:49 on dist-test-slave-8zvk"
server_key: "a3daa75fa8b04d18ff4287cbe2d9bc76"
server_key_iv: "369779b09bae7d3426ec21f6891194d0"
server_key_version: "kuduclusterkey@0"
I20260925 02:58:49.519290 30578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "0d3cf55d13334ed0b020b905e862f2af"
format_stamp: "Formatted at 2026-09-25 02:58:49 on dist-test-slave-8zvk"
server_key: "a3daa75fa8b04d18ff4287cbe2d9bc76"
server_key_iv: "369779b09bae7d3426ec21f6891194d0"
server_key_version: "kuduclusterkey@0"
I20260925 02:58:49.526546 30578 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260925 02:58:49.546336 30593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:58:49.547921 30578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:58:49.548079 30578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "0d3cf55d13334ed0b020b905e862f2af"
format_stamp: "Formatted at 2026-09-25 02:58:49 on dist-test-slave-8zvk"
server_key: "a3daa75fa8b04d18ff4287cbe2d9bc76"
server_key_iv: "369779b09bae7d3426ec21f6891194d0"
server_key_version: "kuduclusterkey@0"
I20260925 02:58:49.548276 30578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:58:49.568859 30578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:58:49.573179 30578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:58:49.573433 30578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:58:49.599022 30578 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:37027
I20260925 02:58:49.599062 30644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:37027 every 8 connection(s)
I20260925 02:58:49.600400 30578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:58:49.604458 30645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:58:49.610118 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 30578
I20260925 02:58:49.610335 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:58:49.611398 19219 external_mini_cluster.cc:1428] Setting key 89f08d75829a6732d568ade1c8f3965c
I20260925 02:58:49.616943 30645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3cf55d13334ed0b020b905e862f2af: Bootstrap starting.
I20260925 02:58:49.620407 30645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d3cf55d13334ed0b020b905e862f2af: Neither blocks nor log segments found. Creating new log.
I20260925 02:58:49.621428 30645 log.cc:824] T 00000000000000000000000000000000 P 0d3cf55d13334ed0b020b905e862f2af: Log is configured to *not* fsync() on all Append() calls
I20260925 02:58:49.624352 30645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3cf55d13334ed0b020b905e862f2af: No bootstrap required, opened a new log
Sep 25 02:58:49 dist-test-slave-8zvk krb5kdc[30132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:58:49.634393 30645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d3cf55d13334ed0b020b905e862f2af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3cf55d13334ed0b020b905e862f2af" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 37027 } }
I20260925 02:58:49.634778 30645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d3cf55d13334ed0b020b905e862f2af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:58:49.634903 30645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d3cf55d13334ed0b020b905e862f2af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3cf55d13334ed0b020b905e862f2af, State: Initialized, Role: FOLLOWER
I20260925 02:58:49.635572 30645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d3cf55d13334ed0b020b905e862f2af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3cf55d13334ed0b020b905e862f2af" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 37027 } }
I20260925 02:58:49.635699 30648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:49.614121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58410 (local address 127.18.196.254:37027)
0925 02:58:49.615443 (+ 1322us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:49.615455 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:49.615514 (+ 59us) server_negotiation.cc:407] Connection header received
0925 02:58:49.617547 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:49.617571 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:49.617942 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:49.618342 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:58:49.619798 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:49.621171 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:49.622532 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:49.623264 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:49.623477 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:49.627714 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:58:49.627721 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:58:49.627734 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:58:49.627772 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:58:49.631305 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:49.632199 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:49.632205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:49.632210 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:49.632368 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:49.632818 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:49.632822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:49.632824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:49.633390 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:58:49.633632 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:49.634086 (+ 454us) server_negotiation.cc:299] Negotiation successful
0925 02:58:49.634446 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1089,"thread_start_us":722,"threads_started":1}
I20260925 02:58:49.635813 30645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d3cf55d13334ed0b020b905e862f2af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:58:49.635946 30645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d3cf55d13334ed0b020b905e862f2af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:58:49.636170 30645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d3cf55d13334ed0b020b905e862f2af [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:58:49.638896 30645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d3cf55d13334ed0b020b905e862f2af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3cf55d13334ed0b020b905e862f2af" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 37027 } }
I20260925 02:58:49.639379 30645 leader_election.cc:302] T 00000000000000000000000000000000 P 0d3cf55d13334ed0b020b905e862f2af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d3cf55d13334ed0b020b905e862f2af; no voters:
I20260925 02:58:49.640007 30645 leader_election.cc:288] T 00000000000000000000000000000000 P 0d3cf55d13334ed0b020b905e862f2af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:58:49.640149 30650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d3cf55d13334ed0b020b905e862f2af [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:58:49.641341 30650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d3cf55d13334ed0b020b905e862f2af [term 1 LEADER]: Becoming Leader. State: Replica: 0d3cf55d13334ed0b020b905e862f2af, State: Running, Role: LEADER
I20260925 02:58:49.642011 30650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d3cf55d13334ed0b020b905e862f2af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3cf55d13334ed0b020b905e862f2af" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 37027 } }
I20260925 02:58:49.642735 30645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d3cf55d13334ed0b020b905e862f2af [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:58:49.645284 30652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3cf55d13334ed0b020b905e862f2af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d3cf55d13334ed0b020b905e862f2af. Latest consensus state: current_term: 1 leader_uuid: "0d3cf55d13334ed0b020b905e862f2af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3cf55d13334ed0b020b905e862f2af" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 37027 } } }
I20260925 02:58:49.645231 30651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3cf55d13334ed0b020b905e862f2af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d3cf55d13334ed0b020b905e862f2af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3cf55d13334ed0b020b905e862f2af" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 37027 } } }
I20260925 02:58:49.645709 30652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3cf55d13334ed0b020b905e862f2af [sys.catalog]: This master's current role is: LEADER
I20260925 02:58:49.645772 30651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3cf55d13334ed0b020b905e862f2af [sys.catalog]: This master's current role is: LEADER
I20260925 02:58:49.646723 30658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:58:49.653183 30658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:58:49.661566 30645 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 02:58:49.662631 30658 catalog_manager.cc:1347] Generated new cluster ID: d3b759c8f36c4af080b9980b0a1a62d3
I20260925 02:58:49.662743 30658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:58:49.686223 30658 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:58:49.688294 30658 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:58:49.697486 30658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d3cf55d13334ed0b020b905e862f2af: Generated new TSK 0
I20260925 02:58:49.699004 30658 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
I20260925 02:58:50.451412 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.196.212:51559/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:37027
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:38657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:58:50 dist-test-slave-8zvk krb5kdc[30132](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790305130, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 02:58:50.929404 30691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:58:50.929850 30691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:58:50.930065 30691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:58:50.942103 30691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:58:50.942373 30691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:58:50.942667 30691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 02:58:50.958976 30691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:38657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.18.196.212:51559/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:37027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:58:50.960675 30691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:58:50.962378 30691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:58:50.979485 30703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:58:50.979621 30705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:58:50.980779 30702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:58:50.981278 30691 server_base.cc:1034] running on GCE node
I20260925 02:58:50.982071 30691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:58:50.983273 30691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:58:50.984530 30691 hybrid_clock.cc:648] HybridClock initialized: now 1790305130984446 us; error 85 us; skew 500 ppm
I20260925 02:58:50.984902 30691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:58:50 dist-test-slave-8zvk krb5kdc[30132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305130, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:58:50.989862 30691 init.cc:377] Logged in from keytab as kudu/127.18.196.193@KRBTEST.COM (short username kudu)
I20260925 02:58:50.991882 30691 webserver.cc:460] Webserver started at http://127.18.196.193:41149/ using document root <none> and password file <none>
I20260925 02:58:50.992614 30691 fs_manager.cc:359] Metadata directory not provided
I20260925 02:58:50.992759 30691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:58:50.993067 30691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:58:50 dist-test-slave-8zvk krb5kdc[30132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305130, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for HTTP/127.18.196.212@KRBTEST.COM
I20260925 02:58:51.012001 30691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "31cf8e2162ba4978bc883f8e240af55f"
format_stamp: "Formatted at 2026-09-25 02:58:51 on dist-test-slave-8zvk"
server_key: "6b95459490fc7917a54dc1b40bd25e41"
server_key_iv: "09676ed1ac5d3f22c495e8ba0d43c50a"
server_key_version: "kuduclusterkey@0"
I20260925 02:58:51.013140 30691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "31cf8e2162ba4978bc883f8e240af55f"
format_stamp: "Formatted at 2026-09-25 02:58:51 on dist-test-slave-8zvk"
server_key: "6b95459490fc7917a54dc1b40bd25e41"
server_key_iv: "09676ed1ac5d3f22c495e8ba0d43c50a"
server_key_version: "kuduclusterkey@0"
I20260925 02:58:51.033113 30691 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.011s sys 0.007s
I20260925 02:58:51.068620 30712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:58:51.071663 30691 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.013s sys 0.003s
I20260925 02:58:51.072085 30691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "31cf8e2162ba4978bc883f8e240af55f"
format_stamp: "Formatted at 2026-09-25 02:58:51 on dist-test-slave-8zvk"
server_key: "6b95459490fc7917a54dc1b40bd25e41"
server_key_iv: "09676ed1ac5d3f22c495e8ba0d43c50a"
server_key_version: "kuduclusterkey@0"
I20260925 02:58:51.072487 30691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:58:51.138150 30691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:58:51.143934 30691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:58:51.144847 30691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:58:51.152336 30691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:58:51.160442 30691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:58:51.160591 30691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260925 02:58:51.160800 30691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:58:51.160918 30691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:58:51 dist-test-slave-8zvk krb5kdc[30132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305130, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for HTTP/127.18.196.212@KRBTEST.COM
I20260925 02:58:51.533919 30824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:43535 every 8 connection(s)
I20260925 02:58:51.534296 30691 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:43535
I20260925 02:58:51.536077 30691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 02:58:51.538378 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 30691
I20260925 02:58:51.538616 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 02:58:51.539548 19219 external_mini_cluster.cc:1428] Setting key 41bf6fbebad6533d8f67eb9e21f8746b
Sep 25 02:58:51 dist-test-slave-8zvk krb5kdc[30132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305130, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:58:51.599112 30828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:51.546964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:42237 (local address 127.18.196.254:37027)
0925 02:58:51.572510 (+ 25546us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:51.572518 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:51.577773 (+ 5255us) server_negotiation.cc:407] Connection header received
0925 02:58:51.577853 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:51.577861 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:51.577970 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:51.578079 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:58:51.581785 (+ 3706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:51.582997 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:51.585425 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:51.586052 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:51.586164 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:51.591243 (+ 5079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:58:51.591248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:58:51.591253 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:58:51.591300 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:58:51.595318 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:51.596273 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:51.596279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:51.596282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:51.596419 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:51.596940 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:51.596945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:51.596948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:51.597327 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:58:51.597466 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:51.598452 (+ 986us) server_negotiation.cc:299] Negotiation successful
0925 02:58:51.598704 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24099,"thread_start_us":24249,"threads_started":1}
I20260925 02:58:51.601938 30827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:51.546227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:37027 (local address 127.18.196.193:42237)
0925 02:58:51.573643 (+ 27416us) negotiation.cc:107] Waiting for socket to connect
0925 02:58:51.573702 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 02:58:51.576505 (+ 2803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:58:51.579598 (+ 3093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:58:51.579611 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:58:51.580648 (+ 1037us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:58:51.581569 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:51.581598 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:51.583214 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:51.583219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:51.585251 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:51.585267 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:51.586526 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:51.586533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:51.586812 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:51.587762 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:58:51.587790 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:58:51.590967 (+ 3177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:58:51.595519 (+ 4552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:58:51.595527 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:58:51.595544 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:58:51.596117 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:58:51.596564 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:58:51.596568 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:58:51.596572 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:58:51.596803 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:58:51.597565 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:58:51.597573 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:58:51.598003 (+ 430us) client_negotiation.cc:769] Sending connection context
0925 02:58:51.599646 (+ 1643us) client_negotiation.cc:240] Negotiation successful
0925 02:58:51.600216 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27017,"thread_start_us":30639,"threads_started":1}
I20260925 02:58:51.604357 30825 heartbeater.cc:347] Connected to a master server at 127.18.196.254:37027
I20260925 02:58:51.604768 30825 heartbeater.cc:464] Registering TS with master...
I20260925 02:58:51.606055 30825 heartbeater.cc:511] Master 127.18.196.254:37027 requested a full tablet report, sending...
I20260925 02:58:51.608960 30610 ts_manager.cc:194] Registered new tserver with Master: 31cf8e2162ba4978bc883f8e240af55f (127.18.196.193:43535)
WARNING: no policy specified for kudu/127.18.196.194@KRBTEST.COM; defaulting to no policy
I20260925 02:58:51.611181 30610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.193@KRBTEST.COM'} at 127.18.196.193:42237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
W20260925 02:58:51.653815 19219 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.18.196.194: real 0.111s user 0.002s sys 0.015s
I20260925 02:58:51.654713 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.196.212:51559/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:37027
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:38657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:58:51 dist-test-slave-8zvk krb5kdc[30132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305130, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for HTTP/127.18.196.212@KRBTEST.COM
W20260925 02:58:52.146744 30832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:58:52.147094 30832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:58:52.147262 30832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:58:52.155632 30832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:58:52.155745 30832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:58:52.155920 30832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 02:58:52.167387 30832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:38657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.18.196.212:51559/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:37027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:58:52.168761 30832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:58:52.170050 30832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:58:52.186136 30846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:58:52.186123 30849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:58:52.186062 30847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:58:52.186987 30832 server_base.cc:1034] running on GCE node
I20260925 02:58:52.187525 30832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:58:52.188498 30832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:58:52.189724 30832 hybrid_clock.cc:648] HybridClock initialized: now 1790305132189672 us; error 62 us; skew 500 ppm
I20260925 02:58:52.190066 30832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:58:52 dist-test-slave-8zvk krb5kdc[30132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305132, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:58:52.193238 30832 init.cc:377] Logged in from keytab as kudu/127.18.196.194@KRBTEST.COM (short username kudu)
I20260925 02:58:52.194809 30832 webserver.cc:460] Webserver started at http://127.18.196.194:42453/ using document root <none> and password file <none>
I20260925 02:58:52.195533 30832 fs_manager.cc:359] Metadata directory not provided
I20260925 02:58:52.195691 30832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:58:52.196004 30832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:58:52 dist-test-slave-8zvk krb5kdc[30132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305132, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for HTTP/127.18.196.212@KRBTEST.COM
I20260925 02:58:52.210069 30832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "e4b2d746dd4c4e308ac5416413e7ce84"
format_stamp: "Formatted at 2026-09-25 02:58:52 on dist-test-slave-8zvk"
server_key: "840fd3e293b266f946f340fa3583ffab"
server_key_iv: "a03716e9cc4433f0ef3d3e217fd4d528"
server_key_version: "kuduclusterkey@0"
I20260925 02:58:52.211472 30832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "e4b2d746dd4c4e308ac5416413e7ce84"
format_stamp: "Formatted at 2026-09-25 02:58:52 on dist-test-slave-8zvk"
server_key: "840fd3e293b266f946f340fa3583ffab"
server_key_iv: "a03716e9cc4433f0ef3d3e217fd4d528"
server_key_version: "kuduclusterkey@0"
I20260925 02:58:52.218598 30832 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 02:58:52.233906 30856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:58:52.235553 30832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:58:52.235749 30832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "e4b2d746dd4c4e308ac5416413e7ce84"
format_stamp: "Formatted at 2026-09-25 02:58:52 on dist-test-slave-8zvk"
server_key: "840fd3e293b266f946f340fa3583ffab"
server_key_iv: "a03716e9cc4433f0ef3d3e217fd4d528"
server_key_version: "kuduclusterkey@0"
I20260925 02:58:52.235992 30832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:58:52.255522 30832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:58:52.260383 30832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:58:52.260703 30832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:58:52.261777 30832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:58:52.263796 30832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:58:52.263901 30832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:58:52.264063 30832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:58:52.264137 30832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:58:52.318356 30832 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:40387
I20260925 02:58:52.318356 30969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:40387 every 8 connection(s)
I20260925 02:58:52.319878 30832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 02:58:52.329180 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 30832
I20260925 02:58:52.329357 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 02:58:52.329870 19219 external_mini_cluster.cc:1428] Setting key ae25f9c8b9984cd36cd96ad01fa9d581
Sep 25 02:58:52 dist-test-slave-8zvk krb5kdc[30132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305132, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:58:52.345460 30972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:52.322904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:41503 (local address 127.18.196.254:37027)
0925 02:58:52.323384 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:52.323388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:52.324751 (+ 1363us) server_negotiation.cc:407] Connection header received
0925 02:58:52.326869 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:52.326874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:52.326979 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:52.327098 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:58:52.328520 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:52.329571 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:52.332115 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:52.332925 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:52.333063 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:52.337578 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:58:52.337585 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:58:52.337590 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:58:52.337631 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:58:52.341786 (+ 4155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:52.342555 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:52.342561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:52.342566 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:52.342753 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:52.343313 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:52.343318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:52.343321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:52.343757 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:58:52.343905 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:52.344745 (+ 840us) server_negotiation.cc:299] Negotiation successful
0925 02:58:52.345014 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":261,"threads_started":1}
I20260925 02:58:52.346379 30973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:52.323602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:37027 (local address 127.18.196.194:41503)
0925 02:58:52.324578 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0925 02:58:52.324619 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:58:52.326570 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:58:52.327347 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:58:52.327355 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:58:52.327770 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:58:52.328342 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:52.328360 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:52.329772 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:52.329778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:52.331948 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:52.331973 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:52.333180 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:52.333187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:52.333463 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:52.334253 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:58:52.334282 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:58:52.337317 (+ 3035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:58:52.341928 (+ 4611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:58:52.341936 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:58:52.341950 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:58:52.342415 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:58:52.342963 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:58:52.342969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:58:52.342972 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:58:52.343187 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:58:52.343910 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:58:52.343920 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:58:52.344266 (+ 346us) client_negotiation.cc:769] Sending connection context
0925 02:58:52.344566 (+ 300us) client_negotiation.cc:240] Negotiation successful
0925 02:58:52.345160 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":541,"threads_started":1}
I20260925 02:58:52.348121 30970 heartbeater.cc:347] Connected to a master server at 127.18.196.254:37027
I20260925 02:58:52.348500 30970 heartbeater.cc:464] Registering TS with master...
I20260925 02:58:52.349254 30970 heartbeater.cc:511] Master 127.18.196.254:37027 requested a full tablet report, sending...
I20260925 02:58:52.350701 30610 ts_manager.cc:194] Registered new tserver with Master: e4b2d746dd4c4e308ac5416413e7ce84 (127.18.196.194:40387)
I20260925 02:58:52.351622 30610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.194@KRBTEST.COM'} at 127.18.196.194:41503
WARNING: no policy specified for kudu/127.18.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
I20260925 02:58:52.398203 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.196.212:51559/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:37027
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:38657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:58:52.614771 30825 heartbeater.cc:503] Master 127.18.196.254:37027 was elected leader, sending a full tablet report...
W20260925 02:58:52.829970 30976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:58:52.830227 30976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:58:52.830466 30976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:58:52.839160 30976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:58:52.839305 30976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:58:52.839498 30976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 02:58:52.850319 30976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:38657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.18.196.212:51559/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:37027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:58:52.851686 30976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:58:52.853011 30976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:58:52.866421 30981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:58:52.866398 30984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:58:52.866456 30982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:58:52.868029 30976 server_base.cc:1034] running on GCE node
I20260925 02:58:52.868628 30976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:58:52.869673 30976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:58:52.871021 30976 hybrid_clock.cc:648] HybridClock initialized: now 1790305132870905 us; error 47 us; skew 500 ppm
I20260925 02:58:52.871438 30976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:58:52 dist-test-slave-8zvk krb5kdc[30132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305132, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:58:52.874805 30976 init.cc:377] Logged in from keytab as kudu/127.18.196.195@KRBTEST.COM (short username kudu)
I20260925 02:58:52.876938 30976 webserver.cc:460] Webserver started at http://127.18.196.195:46787/ using document root <none> and password file <none>
I20260925 02:58:52.877650 30976 fs_manager.cc:359] Metadata directory not provided
I20260925 02:58:52.877784 30976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:58:52.878154 30976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:58:52 dist-test-slave-8zvk krb5kdc[30132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305132, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for HTTP/127.18.196.212@KRBTEST.COM
I20260925 02:58:52.892766 30976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "8a8c1ce6d87a4fa0adc0cff92af8d465"
format_stamp: "Formatted at 2026-09-25 02:58:52 on dist-test-slave-8zvk"
server_key: "b6b2b95ece4d17a2d8811fd97d65188d"
server_key_iv: "9f0e71ee688274596ee7c384b1fb409f"
server_key_version: "kuduclusterkey@0"
I20260925 02:58:52.893735 30976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "8a8c1ce6d87a4fa0adc0cff92af8d465"
format_stamp: "Formatted at 2026-09-25 02:58:52 on dist-test-slave-8zvk"
server_key: "b6b2b95ece4d17a2d8811fd97d65188d"
server_key_iv: "9f0e71ee688274596ee7c384b1fb409f"
server_key_version: "kuduclusterkey@0"
I20260925 02:58:52.900117 30976 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260925 02:58:52.915336 30991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:58:52.916983 30976 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 02:58:52.917212 30976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "8a8c1ce6d87a4fa0adc0cff92af8d465"
format_stamp: "Formatted at 2026-09-25 02:58:52 on dist-test-slave-8zvk"
server_key: "b6b2b95ece4d17a2d8811fd97d65188d"
server_key_iv: "9f0e71ee688274596ee7c384b1fb409f"
server_key_version: "kuduclusterkey@0"
I20260925 02:58:52.917542 30976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:58:52.939932 30976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:58:52.944200 30976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:58:52.944779 30976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:58:52.945960 30976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:58:52.947614 30976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:58:52.947710 30976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:58:52.947930 30976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:58:52.948051 30976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:58:52.998520 30976 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:40189
I20260925 02:58:52.998566 31103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:40189 every 8 connection(s)
I20260925 02:58:52.999992 30976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 02:58:53.001482 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 30976
I20260925 02:58:53.001734 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 02:58:53.002375 19219 external_mini_cluster.cc:1428] Setting key 9c989374e4673d88f2ab35f3574f32a7
Sep 25 02:58:53 dist-test-slave-8zvk krb5kdc[30132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305132, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:58:53.021077 31106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:53.003011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:54727 (local address 127.18.196.254:37027)
0925 02:58:53.003708 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:53.003712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:53.005333 (+ 1621us) server_negotiation.cc:407] Connection header received
0925 02:58:53.007484 (+ 2151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:53.007490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:53.007580 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:53.007687 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:58:53.009263 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.009951 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.011595 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.012019 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.012108 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:53.015403 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:58:53.015407 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:58:53.015410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:58:53.015438 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:58:53.018658 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:53.019341 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:53.019345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:53.019347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:53.019461 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:53.019803 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:53.019806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:53.019808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:53.020051 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:58:53.020153 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:53.020714 (+ 561us) server_negotiation.cc:299] Negotiation successful
0925 02:58:53.020855 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":391,"threads_started":1}
I20260925 02:58:53.022249 31107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:53.003766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:37027 (local address 127.18.196.195:54727)
0925 02:58:53.005131 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0925 02:58:53.005180 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 02:58:53.007226 (+ 2046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:58:53.007880 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:58:53.007888 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:58:53.008493 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:58:53.009123 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:53.009139 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.010113 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.010117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:53.011465 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:53.011477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.012196 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.012201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:53.012373 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:53.012971 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:58:53.012992 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:58:53.015221 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:58:53.018766 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:58:53.018771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:58:53.018781 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:58:53.019242 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:58:53.019546 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:58:53.019549 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:58:53.019551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:58:53.019715 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:58:53.020150 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:58:53.020155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:58:53.020402 (+ 247us) client_negotiation.cc:769] Sending connection context
0925 02:58:53.020607 (+ 205us) client_negotiation.cc:240] Negotiation successful
0925 02:58:53.020914 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"thread_start_us":711,"threads_started":1}
I20260925 02:58:53.023633 31104 heartbeater.cc:347] Connected to a master server at 127.18.196.254:37027
I20260925 02:58:53.023916 31104 heartbeater.cc:464] Registering TS with master...
I20260925 02:58:53.024649 31104 heartbeater.cc:511] Master 127.18.196.254:37027 requested a full tablet report, sending...
I20260925 02:58:53.026101 30610 ts_manager.cc:194] Registered new tserver with Master: 8a8c1ce6d87a4fa0adc0cff92af8d465 (127.18.196.195:40189)
I20260925 02:58:53.026824 30610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.195@KRBTEST.COM'} at 127.18.196.195:54727
I20260925 02:58:53.029716 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:58:53.047663 31106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:53.034651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34684 (local address 127.18.196.254:37027)
0925 02:58:53.034914 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:53.034922 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:53.035239 (+ 317us) server_negotiation.cc:407] Connection header received
0925 02:58:53.035379 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:53.035383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:53.035448 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:53.035522 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:58:53.036842 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.037523 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.038912 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.039396 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.039608 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:53.041803 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:58:53.041808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:58:53.041811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:58:53.041839 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:58:53.044703 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:53.045312 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:53.045317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:53.045321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:53.045457 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:53.045857 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:53.045865 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:53.045868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:53.046229 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:58:53.046578 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:53.046990 (+ 412us) server_negotiation.cc:299] Negotiation successful
0925 02:58:53.047297 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 02:58:53.051638 30610 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:58:53.058987 30610 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:58:53.087881 31118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:53.072134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:40189 (local address 127.0.0.1:39120)
0925 02:58:53.073563 (+ 1429us) negotiation.cc:107] Waiting for socket to connect
0925 02:58:53.073629 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 02:58:53.073836 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:58:53.076877 (+ 3041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:58:53.076887 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:58:53.076921 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:58:53.077991 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:53.078007 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.080532 (+ 2525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.080538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:53.083964 (+ 3426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:53.083980 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.086268 (+ 2288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.086276 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:53.086656 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:53.086721 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 02:58:53.086879 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 02:58:53.087119 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1135,"mutex_wait_us":240,"thread_start_us":2072,"threads_started":1}
I20260925 02:58:53.088661 31117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:53.072175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:43535 (local address 127.0.0.1:50562)
0925 02:58:53.074447 (+ 2272us) negotiation.cc:107] Waiting for socket to connect
0925 02:58:53.074466 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:58:53.074579 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:58:53.077726 (+ 3147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:58:53.077733 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:58:53.077763 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:58:53.078861 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:53.078879 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.081286 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.081294 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:53.084629 (+ 3335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:53.084648 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.087680 (+ 3032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.087688 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:53.087937 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:53.087974 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 02:58:53.088068 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 02:58:53.088205 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2089,"mutex_wait_us":147,"thread_start_us":1477,"threads_started":1}
I20260925 02:58:53.089027 31119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:53.072480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39120 (local address 127.18.196.195:40189)
0925 02:58:53.073713 (+ 1233us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:53.073720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:53.073756 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:58:53.076338 (+ 2582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:53.076346 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:53.076590 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:53.077452 (+ 862us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:58:53.078424 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.080423 (+ 1999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.084154 (+ 3731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.085822 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.086149 (+ 327us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:53.086410 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:53.087000 (+ 590us) server_negotiation.cc:299] Negotiation successful
0925 02:58:53.087772 (+ 772us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1038,"thread_start_us":1299,"threads_started":1}
I20260925 02:58:53.089823 31121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:53.073325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:40387 (local address 127.0.0.1:55152)
0925 02:58:53.075825 (+ 2500us) negotiation.cc:107] Waiting for socket to connect
0925 02:58:53.075859 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 02:58:53.076042 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:58:53.077200 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:58:53.077207 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:58:53.077237 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:58:53.078962 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:53.078975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.080476 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.080482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:53.084340 (+ 3858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:53.084353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.088709 (+ 4356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.088718 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:53.088893 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:53.088940 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 02:58:53.089018 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 02:58:53.089163 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2366,"mutex_wait_us":688,"thread_start_us":2225,"threads_started":1}
I20260925 02:58:53.090377 31120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:53.073088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50562 (local address 127.18.196.193:43535)
0925 02:58:53.077184 (+ 4096us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:53.077190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:53.077211 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:58:53.077295 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:53.077300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:53.077506 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:53.077744 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:58:53.079079 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.081092 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.084845 (+ 3753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.087362 (+ 2517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.089258 (+ 1896us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:53.089426 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:53.089523 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 02:58:53.089744 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3944,"thread_start_us":3667,"threads_started":1}
I20260925 02:58:53.093760 31122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:53.073631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55152 (local address 127.18.196.194:40387)
0925 02:58:53.076397 (+ 2766us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:53.076406 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:53.076430 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:58:53.076513 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:53.076519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:53.076781 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:53.076959 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:58:53.079159 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.080337 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.084522 (+ 4185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.088442 (+ 3920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.088619 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:53.088798 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:53.093184 (+ 4386us) server_negotiation.cc:299] Negotiation successful
0925 02:58:53.093377 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2546,"thread_start_us":2389,"threads_started":1}
I20260925 02:58:53.094682 30760 tablet_service.cc:1467] Processing CreateTablet for tablet e0440d9f2fea462196eda16df2a846e4 (DEFAULT_TABLE table=test-table [id=8aa9cc1b714d47089c440135bfc28ea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:58:53.095185 31039 tablet_service.cc:1467] Processing CreateTablet for tablet e0440d9f2fea462196eda16df2a846e4 (DEFAULT_TABLE table=test-table [id=8aa9cc1b714d47089c440135bfc28ea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:58:53.096328 30760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0440d9f2fea462196eda16df2a846e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:58:53.096634 31039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0440d9f2fea462196eda16df2a846e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:58:53.096892 30904 tablet_service.cc:1467] Processing CreateTablet for tablet e0440d9f2fea462196eda16df2a846e4 (DEFAULT_TABLE table=test-table [id=8aa9cc1b714d47089c440135bfc28ea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:58:53.097880 30904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0440d9f2fea462196eda16df2a846e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:58:53.111739 31123 tablet_bootstrap.cc:492] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465: Bootstrap starting.
I20260925 02:58:53.113618 31125 tablet_bootstrap.cc:492] T e0440d9f2fea462196eda16df2a846e4 P e4b2d746dd4c4e308ac5416413e7ce84: Bootstrap starting.
I20260925 02:58:53.115278 31123 tablet_bootstrap.cc:654] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465: Neither blocks nor log segments found. Creating new log.
I20260925 02:58:53.116307 31123 log.cc:824] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465: Log is configured to *not* fsync() on all Append() calls
I20260925 02:58:53.117328 31124 tablet_bootstrap.cc:492] T e0440d9f2fea462196eda16df2a846e4 P 31cf8e2162ba4978bc883f8e240af55f: Bootstrap starting.
I20260925 02:58:53.117846 31125 tablet_bootstrap.cc:654] T e0440d9f2fea462196eda16df2a846e4 P e4b2d746dd4c4e308ac5416413e7ce84: Neither blocks nor log segments found. Creating new log.
I20260925 02:58:53.118752 31125 log.cc:824] T e0440d9f2fea462196eda16df2a846e4 P e4b2d746dd4c4e308ac5416413e7ce84: Log is configured to *not* fsync() on all Append() calls
I20260925 02:58:53.120633 31124 tablet_bootstrap.cc:654] T e0440d9f2fea462196eda16df2a846e4 P 31cf8e2162ba4978bc883f8e240af55f: Neither blocks nor log segments found. Creating new log.
I20260925 02:58:53.122290 31125 tablet_bootstrap.cc:492] T e0440d9f2fea462196eda16df2a846e4 P e4b2d746dd4c4e308ac5416413e7ce84: No bootstrap required, opened a new log
I20260925 02:58:53.122557 31125 ts_tablet_manager.cc:1397] T e0440d9f2fea462196eda16df2a846e4 P e4b2d746dd4c4e308ac5416413e7ce84: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260925 02:58:53.122638 31124 log.cc:824] T e0440d9f2fea462196eda16df2a846e4 P 31cf8e2162ba4978bc883f8e240af55f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:58:53.129079 31124 tablet_bootstrap.cc:492] T e0440d9f2fea462196eda16df2a846e4 P 31cf8e2162ba4978bc883f8e240af55f: No bootstrap required, opened a new log
I20260925 02:58:53.129375 31124 ts_tablet_manager.cc:1397] T e0440d9f2fea462196eda16df2a846e4 P 31cf8e2162ba4978bc883f8e240af55f: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260925 02:58:53.130174 31123 tablet_bootstrap.cc:492] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465: No bootstrap required, opened a new log
I20260925 02:58:53.130564 31123 ts_tablet_manager.cc:1397] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s
I20260925 02:58:53.130538 31125 raft_consensus.cc:348] T e0440d9f2fea462196eda16df2a846e4 P e4b2d746dd4c4e308ac5416413e7ce84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a8c1ce6d87a4fa0adc0cff92af8d465" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 40189 } } peers { permanent_uuid: "31cf8e2162ba4978bc883f8e240af55f" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43535 } } peers { permanent_uuid: "e4b2d746dd4c4e308ac5416413e7ce84" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40387 } }
I20260925 02:58:53.130941 31125 raft_consensus.cc:374] T e0440d9f2fea462196eda16df2a846e4 P e4b2d746dd4c4e308ac5416413e7ce84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:58:53.131035 31125 raft_consensus.cc:729] T e0440d9f2fea462196eda16df2a846e4 P e4b2d746dd4c4e308ac5416413e7ce84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b2d746dd4c4e308ac5416413e7ce84, State: Initialized, Role: FOLLOWER
I20260925 02:58:53.131688 31125 consensus_queue.cc:260] T e0440d9f2fea462196eda16df2a846e4 P e4b2d746dd4c4e308ac5416413e7ce84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a8c1ce6d87a4fa0adc0cff92af8d465" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 40189 } } peers { permanent_uuid: "31cf8e2162ba4978bc883f8e240af55f" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43535 } } peers { permanent_uuid: "e4b2d746dd4c4e308ac5416413e7ce84" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40387 } }
I20260925 02:58:53.139680 30970 heartbeater.cc:503] Master 127.18.196.254:37027 was elected leader, sending a full tablet report...
I20260925 02:58:53.140198 31125 ts_tablet_manager.cc:1428] T e0440d9f2fea462196eda16df2a846e4 P e4b2d746dd4c4e308ac5416413e7ce84: Time spent starting tablet: real 0.017s user 0.005s sys 0.014s
I20260925 02:58:53.142964 31124 raft_consensus.cc:348] T e0440d9f2fea462196eda16df2a846e4 P 31cf8e2162ba4978bc883f8e240af55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a8c1ce6d87a4fa0adc0cff92af8d465" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 40189 } } peers { permanent_uuid: "31cf8e2162ba4978bc883f8e240af55f" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43535 } } peers { permanent_uuid: "e4b2d746dd4c4e308ac5416413e7ce84" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40387 } }
I20260925 02:58:53.143707 31124 raft_consensus.cc:374] T e0440d9f2fea462196eda16df2a846e4 P 31cf8e2162ba4978bc883f8e240af55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:58:53.143929 31124 raft_consensus.cc:729] T e0440d9f2fea462196eda16df2a846e4 P 31cf8e2162ba4978bc883f8e240af55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31cf8e2162ba4978bc883f8e240af55f, State: Initialized, Role: FOLLOWER
I20260925 02:58:53.144892 31124 consensus_queue.cc:260] T e0440d9f2fea462196eda16df2a846e4 P 31cf8e2162ba4978bc883f8e240af55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a8c1ce6d87a4fa0adc0cff92af8d465" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 40189 } } peers { permanent_uuid: "31cf8e2162ba4978bc883f8e240af55f" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43535 } } peers { permanent_uuid: "e4b2d746dd4c4e308ac5416413e7ce84" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40387 } }
I20260925 02:58:53.145752 31123 raft_consensus.cc:348] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a8c1ce6d87a4fa0adc0cff92af8d465" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 40189 } } peers { permanent_uuid: "31cf8e2162ba4978bc883f8e240af55f" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43535 } } peers { permanent_uuid: "e4b2d746dd4c4e308ac5416413e7ce84" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40387 } }
I20260925 02:58:53.146224 31123 raft_consensus.cc:374] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:58:53.146338 31123 raft_consensus.cc:729] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a8c1ce6d87a4fa0adc0cff92af8d465, State: Initialized, Role: FOLLOWER
I20260925 02:58:53.146802 31124 ts_tablet_manager.cc:1428] T e0440d9f2fea462196eda16df2a846e4 P 31cf8e2162ba4978bc883f8e240af55f: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260925 02:58:53.146965 31123 consensus_queue.cc:260] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a8c1ce6d87a4fa0adc0cff92af8d465" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 40189 } } peers { permanent_uuid: "31cf8e2162ba4978bc883f8e240af55f" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43535 } } peers { permanent_uuid: "e4b2d746dd4c4e308ac5416413e7ce84" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40387 } }
I20260925 02:58:53.148036 31104 heartbeater.cc:503] Master 127.18.196.254:37027 was elected leader, sending a full tablet report...
I20260925 02:58:53.148686 31123 ts_tablet_manager.cc:1428] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465: Time spent starting tablet: real 0.018s user 0.013s sys 0.003s
I20260925 02:58:53.237728 31131 raft_consensus.cc:482] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:58:53.238315 31131 raft_consensus.cc:504] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a8c1ce6d87a4fa0adc0cff92af8d465" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 40189 } } peers { permanent_uuid: "31cf8e2162ba4978bc883f8e240af55f" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43535 } } peers { permanent_uuid: "e4b2d746dd4c4e308ac5416413e7ce84" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40387 } }
I20260925 02:58:53.240757 31131 leader_election.cc:288] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31cf8e2162ba4978bc883f8e240af55f (127.18.196.193:43535), e4b2d746dd4c4e308ac5416413e7ce84 (127.18.196.194:40387)
I20260925 02:58:53.247844 31120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:53.241094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:47077 (local address 127.18.196.193:43535)
0925 02:58:53.241317 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:53.241323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:53.241484 (+ 161us) server_negotiation.cc:407] Connection header received
0925 02:58:53.241700 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:53.241704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:53.241764 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:53.241868 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:58:53.242440 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.243478 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.245698 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.247026 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.247205 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:53.247289 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:53.247575 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 02:58:53.247668 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:58:53.247927 31107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:53.241195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:43535 (local address 127.18.196.195:47077)
0925 02:58:53.241351 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 02:58:53.241372 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:58:53.241519 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:58:53.241892 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:58:53.241897 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:58:53.241936 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:58:53.242298 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:53.242314 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.243614 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.243622 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:53.245459 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:53.245475 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.247196 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.247201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:53.247413 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:53.247448 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 02:58:53.247548 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 02:58:53.247675 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 02:58:53.248935 30780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0440d9f2fea462196eda16df2a846e4" candidate_uuid: "8a8c1ce6d87a4fa0adc0cff92af8d465" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31cf8e2162ba4978bc883f8e240af55f" is_pre_election: true
I20260925 02:58:53.249360 31122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:53.241123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:56475 (local address 127.18.196.194:40387)
0925 02:58:53.241409 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:53.241415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:53.242280 (+ 865us) server_negotiation.cc:407] Connection header received
0925 02:58:53.242405 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:53.242411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:53.242525 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:53.242597 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:58:53.243333 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.244567 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.247348 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.248462 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.248672 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:53.248742 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:53.248984 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 02:58:53.249109 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 02:58:53.249470 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:53.241476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:40387 (local address 127.18.196.195:56475)
0925 02:58:53.242180 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0925 02:58:53.242194 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:58:53.242286 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:58:53.242654 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:58:53.242660 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:58:53.242686 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:58:53.243171 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:53.243183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.244722 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.244725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:53.247204 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:58:53.247218 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.248679 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.248685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:58:53.248835 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:53.248857 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:58:53.248958 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 02:58:53.249077 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":344,"threads_started":1}
I20260925 02:58:53.249521 30780 raft_consensus.cc:2454] T e0440d9f2fea462196eda16df2a846e4 P 31cf8e2162ba4978bc883f8e240af55f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a8c1ce6d87a4fa0adc0cff92af8d465 in term 0.
I20260925 02:58:53.250330 30994 leader_election.cc:302] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31cf8e2162ba4978bc883f8e240af55f, 8a8c1ce6d87a4fa0adc0cff92af8d465; no voters:
I20260925 02:58:53.250748 31131 raft_consensus.cc:2781] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:58:53.250725 30924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0440d9f2fea462196eda16df2a846e4" candidate_uuid: "8a8c1ce6d87a4fa0adc0cff92af8d465" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4b2d746dd4c4e308ac5416413e7ce84" is_pre_election: true
I20260925 02:58:53.250999 31131 raft_consensus.cc:482] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:58:53.251252 31131 raft_consensus.cc:3037] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:58:53.251228 30924 raft_consensus.cc:2454] T e0440d9f2fea462196eda16df2a846e4 P e4b2d746dd4c4e308ac5416413e7ce84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a8c1ce6d87a4fa0adc0cff92af8d465 in term 0.
W20260925 02:58:53.252334 31105 tablet.cc:2369] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:58:53.254225 31131 raft_consensus.cc:504] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a8c1ce6d87a4fa0adc0cff92af8d465" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 40189 } } peers { permanent_uuid: "31cf8e2162ba4978bc883f8e240af55f" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43535 } } peers { permanent_uuid: "e4b2d746dd4c4e308ac5416413e7ce84" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40387 } }
I20260925 02:58:53.254933 31131 leader_election.cc:288] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465 [CANDIDATE]: Term 1 election: Requested vote from peers 31cf8e2162ba4978bc883f8e240af55f (127.18.196.193:43535), e4b2d746dd4c4e308ac5416413e7ce84 (127.18.196.194:40387)
I20260925 02:58:53.255379 30780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0440d9f2fea462196eda16df2a846e4" candidate_uuid: "8a8c1ce6d87a4fa0adc0cff92af8d465" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31cf8e2162ba4978bc883f8e240af55f"
I20260925 02:58:53.255462 30924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0440d9f2fea462196eda16df2a846e4" candidate_uuid: "8a8c1ce6d87a4fa0adc0cff92af8d465" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4b2d746dd4c4e308ac5416413e7ce84"
I20260925 02:58:53.255676 30780 raft_consensus.cc:3037] T e0440d9f2fea462196eda16df2a846e4 P 31cf8e2162ba4978bc883f8e240af55f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:58:53.255759 30924 raft_consensus.cc:3037] T e0440d9f2fea462196eda16df2a846e4 P e4b2d746dd4c4e308ac5416413e7ce84 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:58:53.258252 30780 raft_consensus.cc:2454] T e0440d9f2fea462196eda16df2a846e4 P 31cf8e2162ba4978bc883f8e240af55f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a8c1ce6d87a4fa0adc0cff92af8d465 in term 1.
I20260925 02:58:53.258486 30924 raft_consensus.cc:2454] T e0440d9f2fea462196eda16df2a846e4 P e4b2d746dd4c4e308ac5416413e7ce84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a8c1ce6d87a4fa0adc0cff92af8d465 in term 1.
I20260925 02:58:53.258662 30994 leader_election.cc:302] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31cf8e2162ba4978bc883f8e240af55f, 8a8c1ce6d87a4fa0adc0cff92af8d465; no voters:
I20260925 02:58:53.258973 31131 raft_consensus.cc:2781] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:58:53.259364 31131 raft_consensus.cc:686] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465 [term 1 LEADER]: Becoming Leader. State: Replica: 8a8c1ce6d87a4fa0adc0cff92af8d465, State: Running, Role: LEADER
I20260925 02:58:53.259868 31131 consensus_queue.cc:237] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a8c1ce6d87a4fa0adc0cff92af8d465" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 40189 } } peers { permanent_uuid: "31cf8e2162ba4978bc883f8e240af55f" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43535 } } peers { permanent_uuid: "e4b2d746dd4c4e308ac5416413e7ce84" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40387 } }
I20260925 02:58:53.264364 30610 catalog_manager.cc:5495] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a8c1ce6d87a4fa0adc0cff92af8d465 (127.18.196.195). New cstate: current_term: 1 leader_uuid: "8a8c1ce6d87a4fa0adc0cff92af8d465" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a8c1ce6d87a4fa0adc0cff92af8d465" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 40189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31cf8e2162ba4978bc883f8e240af55f" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4b2d746dd4c4e308ac5416413e7ce84" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40387 } health_report { overall_health: UNKNOWN } } }
I20260925 02:58:53.287848 31119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:53.282210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39132 (local address 127.18.196.195:40189)
0925 02:58:53.282376 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:53.282380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:53.282396 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:58:53.282471 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:53.282474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:53.282570 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:53.282692 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:58:53.283180 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.283961 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.285810 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.286237 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.286327 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:53.286783 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:58:53.287351 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:58:53.287452 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:53.287600 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 02:58:53.287662 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:58:53.296008 30924 raft_consensus.cc:1261] T e0440d9f2fea462196eda16df2a846e4 P e4b2d746dd4c4e308ac5416413e7ce84 [term 1 FOLLOWER]: Refusing update from remote peer 8a8c1ce6d87a4fa0adc0cff92af8d465: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:58:53.296038 30780 raft_consensus.cc:1261] T e0440d9f2fea462196eda16df2a846e4 P 31cf8e2162ba4978bc883f8e240af55f [term 1 FOLLOWER]: Refusing update from remote peer 8a8c1ce6d87a4fa0adc0cff92af8d465: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:58:53.297057 31133 consensus_queue.cc:1035] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31cf8e2162ba4978bc883f8e240af55f" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 43535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:58:53.297457 31131 consensus_queue.cc:1035] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4b2d746dd4c4e308ac5416413e7ce84" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:58:53.311709 31136 mvcc.cc:205] Tried to move back new op lower bound from 7333089825969664000 to 7333089825836343296. Current Snapshot: MvccSnapshot[applied={T|T < 7333089825969664000}]
I20260925 02:58:53.314896 31138 mvcc.cc:205] Tried to move back new op lower bound from 7333089825969664000 to 7333089825836343296. Current Snapshot: MvccSnapshot[applied={T|T < 7333089825969664000}]
I20260925 02:58:53.325385 30609 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:58:53.325775 30609 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:58:53.330528 30609 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d3cf55d13334ed0b020b905e862f2af: Sending DeleteTablet for 3 replicas of tablet e0440d9f2fea462196eda16df2a846e4
I20260925 02:58:53.331925 31039 tablet_service.cc:1514] Processing DeleteTablet for tablet e0440d9f2fea462196eda16df2a846e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:58:53 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:39120
I20260925 02:58:53.332247 30760 tablet_service.cc:1514] Processing DeleteTablet for tablet e0440d9f2fea462196eda16df2a846e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:58:53 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:50562
I20260925 02:58:53.335701 30904 tablet_service.cc:1514] Processing DeleteTablet for tablet e0440d9f2fea462196eda16df2a846e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:58:53 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:55152
I20260925 02:58:53.344993 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 30691
I20260925 02:58:53.348323 31145 tablet_replica.cc:330] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465: stopping tablet replica
I20260925 02:58:53.349035 31145 raft_consensus.cc:2229] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:58:53.349897 31145 raft_consensus.cc:2258] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:58:53.351431 31146 tablet_replica.cc:330] T e0440d9f2fea462196eda16df2a846e4 P 31cf8e2162ba4978bc883f8e240af55f: stopping tablet replica
I20260925 02:58:53.352272 31147 tablet_replica.cc:330] T e0440d9f2fea462196eda16df2a846e4 P e4b2d746dd4c4e308ac5416413e7ce84: stopping tablet replica
I20260925 02:58:53.352700 31145 ts_tablet_manager.cc:1905] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:58:53.352828 31146 raft_consensus.cc:2229] T e0440d9f2fea462196eda16df2a846e4 P 31cf8e2162ba4978bc883f8e240af55f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:58:53.353390 31146 raft_consensus.cc:2258] T e0440d9f2fea462196eda16df2a846e4 P 31cf8e2162ba4978bc883f8e240af55f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:58:53.354084 31147 raft_consensus.cc:2229] T e0440d9f2fea462196eda16df2a846e4 P e4b2d746dd4c4e308ac5416413e7ce84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:58:53.354661 31147 raft_consensus.cc:2258] T e0440d9f2fea462196eda16df2a846e4 P e4b2d746dd4c4e308ac5416413e7ce84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:58:53.357095 31146 ts_tablet_manager.cc:1905] T e0440d9f2fea462196eda16df2a846e4 P 31cf8e2162ba4978bc883f8e240af55f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:58:53.358466 31147 ts_tablet_manager.cc:1905] T e0440d9f2fea462196eda16df2a846e4 P e4b2d746dd4c4e308ac5416413e7ce84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:58:53.366142 31146 ts_tablet_manager.cc:1918] T e0440d9f2fea462196eda16df2a846e4 P 31cf8e2162ba4978bc883f8e240af55f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:58:53.366364 31146 log.cc:1196] T e0440d9f2fea462196eda16df2a846e4 P 31cf8e2162ba4978bc883f8e240af55f: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-0/wal/wals/e0440d9f2fea462196eda16df2a846e4
I20260925 02:58:53.366406 31147 ts_tablet_manager.cc:1918] T e0440d9f2fea462196eda16df2a846e4 P e4b2d746dd4c4e308ac5416413e7ce84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:58:53.367100 31146 ts_tablet_manager.cc:1939] T e0440d9f2fea462196eda16df2a846e4 P 31cf8e2162ba4978bc883f8e240af55f: Deleting consensus metadata
I20260925 02:58:53.369150 30596 catalog_manager.cc:4841] TS 31cf8e2162ba4978bc883f8e240af55f (127.18.196.193:43535): tablet e0440d9f2fea462196eda16df2a846e4 (table test-table [id=8aa9cc1b714d47089c440135bfc28ea3]) successfully deleted
I20260925 02:58:53.369601 31147 log.cc:1196] T e0440d9f2fea462196eda16df2a846e4 P e4b2d746dd4c4e308ac5416413e7ce84: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/e0440d9f2fea462196eda16df2a846e4
I20260925 02:58:53.370204 31147 ts_tablet_manager.cc:1939] T e0440d9f2fea462196eda16df2a846e4 P e4b2d746dd4c4e308ac5416413e7ce84: Deleting consensus metadata
I20260925 02:58:53.371480 31145 ts_tablet_manager.cc:1918] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:58:53.371757 31145 log.cc:1196] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/e0440d9f2fea462196eda16df2a846e4
I20260925 02:58:53.372229 30595 catalog_manager.cc:4841] TS e4b2d746dd4c4e308ac5416413e7ce84 (127.18.196.194:40387): tablet e0440d9f2fea462196eda16df2a846e4 (table test-table [id=8aa9cc1b714d47089c440135bfc28ea3]) successfully deleted
I20260925 02:58:53.373059 31145 ts_tablet_manager.cc:1939] T e0440d9f2fea462196eda16df2a846e4 P 8a8c1ce6d87a4fa0adc0cff92af8d465: Deleting consensus metadata
Sep 25 02:58:53 dist-test-slave-8zvk krb5kdc[30132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.193@KRBTEST.COM
I20260925 02:58:53.375296 30594 catalog_manager.cc:4841] TS 8a8c1ce6d87a4fa0adc0cff92af8d465 (127.18.196.195:40189): tablet e0440d9f2fea462196eda16df2a846e4 (table test-table [id=8aa9cc1b714d47089c440135bfc28ea3]) successfully deleted
I20260925 02:58:53.381013 31120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:53.346319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50564 (local address 127.18.196.193:43535)
0925 02:58:53.346499 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:53.346508 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:53.362234 (+ 15726us) server_negotiation.cc:407] Connection header received
0925 02:58:53.362448 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:53.362456 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:53.362559 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:53.362676 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:58:53.364566 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.365712 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.367753 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.368442 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.368583 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:53.374342 (+ 5759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:58:53.374355 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:58:53.374368 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:58:53.374405 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:58:53.377999 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:53.378739 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:53.378746 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:53.378749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:53.378981 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:53.379380 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:53.379385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:53.379388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:53.379767 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:58:53.380077 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:53.380386 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 02:58:53.380642 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 02:58:53.381922 30820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:58:53.484412 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 30691
I20260925 02:58:53.502331 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 30832
Sep 25 02:58:53 dist-test-slave-8zvk krb5kdc[30132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.194@KRBTEST.COM
I20260925 02:58:53.519193 31122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:53.503205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55162 (local address 127.18.196.194:40387)
0925 02:58:53.503407 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:53.503414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:53.503474 (+ 60us) server_negotiation.cc:407] Connection header received
0925 02:58:53.503665 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:53.503672 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:53.503761 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:53.503875 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:58:53.505129 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.506273 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.507669 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.508283 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.508414 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:53.512420 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:58:53.512440 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:58:53.512468 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:58:53.512514 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:58:53.516262 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:53.517027 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:53.517036 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:53.517040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:53.517208 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:53.517543 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:53.517550 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:53.517554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:53.518033 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:58:53.518355 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:53.518523 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 02:58:53.518736 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 02:58:53.519891 30965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:58:53.614039 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 30832
I20260925 02:58:53.637018 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 30976
Sep 25 02:58:53 dist-test-slave-8zvk krb5kdc[30132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.195@KRBTEST.COM
I20260925 02:58:53.656383 31119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:58:53.637844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39146 (local address 127.18.196.195:40189)
0925 02:58:53.638073 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 02:58:53.638078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:58:53.638096 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:58:53.638302 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:58:53.638307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:58:53.638375 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:58:53.638464 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:58:53.640336 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.641114 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.643097 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:58:53.643601 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:58:53.643702 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:58:53.649338 (+ 5636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:58:53.649354 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:58:53.649373 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:58:53.649417 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:58:53.653444 (+ 4027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:53.654097 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:53.654109 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:53.654119 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:53.654334 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:58:53.654723 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:58:53.654728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:58:53.654732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:58:53.655214 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:58:53.655519 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0925 02:58:53.655752 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 02:58:53.655982 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260925 02:58:53.657222 31099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:58:53.776300 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 30976
I20260925 02:58:53.796020 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 30578
I20260925 02:58:53.797196 30640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:58:53.908700 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 30578
I20260925 02:58:53.929804 19219 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 02:58:54.265045 19219 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 02:58:54.265374 19219 mini_ranger.cc:66] Stopping Ranger...
I20260925 02:58:54.610517 19219 mini_ranger.cc:68] Stopped Ranger
2026-09-25 02:58:54.610 UTC [30162] LOG: received smart shutdown request
2026-09-25 02:58:54.614 UTC [30162] LOG: background worker "logical replication launcher" (PID 30171) exited with exit code 1
2026-09-25 02:58:54.614 UTC [30166] LOG: shutting down
2026-09-25 02:58:54.770 UTC [30162] LOG: database system is shut down
2026-09-25T02:58:54Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50738 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:58:55 dist-test-slave-8zvk krb5kdc[31161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:58:55 dist-test-slave-8zvk krb5kdc[31161](info): set up 2 sockets
Sep 25 02:58:55 dist-test-slave-8zvk krb5kdc[31161](info): commencing operation
krb5kdc: starting...
W20260925 02:58:56.006376 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 02:58:56 dist-test-slave-8zvk krb5kdc[31161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:58:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:58:56Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.18.196.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.18.196.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.18.196.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/rangeradmin_127.18.196.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.18.196.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/rangeradmin_127.18.196.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.18.196.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.18.196.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.18.196.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/rangerlookup_127.18.196.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.18.196.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/rangerlookup_127.18.196.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.196.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.196.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.196.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/HTTP_127.18.196.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.196.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/HTTP_127.18.196.212@KRBTEST.COM.keytab.
I20260925 02:58:56.330454 19219 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskM3GXH0/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/postgres -l logfile start
2026-09-25 02:58:58.734 UTC [31191] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 02:58:58.734 UTC [31191] LOG: listening on IPv4 address "127.18.196.212", port 48915
2026-09-25 02:58:58.737 UTC [31191] LOG: listening on Unix socket "/tmp/.s.PGSQL.48915"
2026-09-25 02:58:58.759 UTC [31194] LOG: database system was shut down at 2026-09-25 02:58:57 UTC
2026-09-25 02:58:58.762 UTC [31191] LOG: database system is ready to accept connections
I20260925 02:58:59.726068 19219 mini_postgres.cc:96] Postgres bound to 48915
2026-09-25 02:58:59.743 UTC [31202] FATAL: database "slave" does not exist
127.18.196.212:48915 - accepting connections
I20260925 02:58:59.744256 19219 mini_ranger.cc:161] Starting Ranger...
I20260925 02:58:59.767704 19219 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 02:59:00.224754 19219 mini_ranger.cc:87] Created ranger Postgres database
I20260925 02:59:00.224879 19219 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ranger-admin
2026-09-25 02:59:00,504 [I] DB FLAVOR :POSTGRES
2026-09-25 02:59:00,505 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:59:00,505 [I] Checking connection..
2026-09-25 02:59:00,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:48915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 02:59:00,748 [I] Checking connection passed.
2026-09-25 02:59:00,749 [I] --------- Verifying version history table ---------
2026-09-25 02:59:00,749 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:48915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:59:00,983 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:59:00,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:48915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:59:01,231 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:59:01,231 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 02:59:01,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:48915/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 02:59:01.477 UTC [31266] WARNING: there is no transaction in progress
2026-09-25 02:59:01,483 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 02:59:01,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:48915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:59:01,725 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 02:59:01,725 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:59:01,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:48915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 02:59:01,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:48915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 02:59:02,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:48915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8zvk.c.gcp-upstream.internal','N') ;"
2026-09-25 02:59:02,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:48915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:59:02,734 [I] Table x_portal_user does not exist in database ranger
2026-09-25 02:59:02,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:48915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:59:02,973 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 02:59:02,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:48915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:59:03,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:48915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:59:03,459 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 02:59:03,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:48915/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 02:59:04.395 UTC [31374] WARNING: there is no transaction in progress
2026-09-25 02:59:04.410 UTC [31374] WARNING: there is no transaction in progress
2026-09-25 02:59:04.425 UTC [31374] WARNING: there is no transaction in progress
2026-09-25 02:59:04.439 UTC [31374] WARNING: there is no transaction in progress
2026-09-25 02:59:04.457 UTC [31374] WARNING: there is no transaction in progress
2026-09-25 02:59:04.476 UTC [31374] WARNING: there is no transaction in progress
2026-09-25 02:59:04.536 UTC [31374] WARNING: there is no transaction in progress
2026-09-25 02:59:04.543 UTC [31374] WARNING: there is no transaction in progress
2026-09-25 02:59:04.556 UTC [31374] WARNING: there is no transaction in progress
2026-09-25 02:59:04.566 UTC [31374] WARNING: there is no transaction in progress
2026-09-25 02:59:04.576 UTC [31374] WARNING: there is no transaction in progress
2026-09-25 02:59:04.589 UTC [31374] WARNING: there is no transaction in progress
2026-09-25 02:59:04.599 UTC [31374] WARNING: there is no transaction in progress
2026-09-25 02:59:04.605 UTC [31374] WARNING: there is no transaction in progress
2026-09-25 02:59:04.611 UTC [31374] WARNING: there is no transaction in progress
2026-09-25 02:59:04.617 UTC [31374] WARNING: there is no transaction in progress
2026-09-25 02:59:05,218 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 02:59:05,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:48915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 02:59:05,463 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 02:59:05,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:48915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:59:05,708 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 02:59:05,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:48915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:59:05,948 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 02:59:05,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:48915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:59:06,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:48915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:59:06,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:48915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8zvk.c.gcp-upstream.internal';"
2026-09-25 02:59:06,669 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 02:59:06,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:48915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 02:59:06,923 [I] DB_PATCHES have already been applied
I20260925 02:59:06.929617 19219 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ranger-admin:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskM3GXH0/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 02:59:06.929695 19219 mini_ranger.cc:193] Using host: 127.18.196.212
I20260925 02:59:06.945176 19219 mini_ranger.cc:227] Ranger admin URL: http://127.18.196.212:49545
Sep 25, 2026 2:59:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:59:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:59:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:59:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:59:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:59:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:59:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 02:59:08.065399 19219 mini_ranger.cc:160] Time spent starting Ranger: real 8.321s user 0.004s sys 0.033s
Sep 25, 2026 2:59:27 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:59:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 2:59:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:59:27 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:59:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:59:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:59:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:59:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:59:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:59:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:59:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:59:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:59:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:59:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:59:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:59:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:59:27 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:59:27 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 02:59:28.565738 19219 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.18.196.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.18.196.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.18.196.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/rangerkms_127.18.196.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.18.196.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/rangerkms_127.18.196.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.196.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.18.196.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.196.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/HTTP_127.18.196.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.196.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/HTTP_127.18.196.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 02:59:28 dist-test-slave-8zvk krb5kdc[31161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305168, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 02:59:28.732647 19219 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 02:59:28.752661 19219 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 02:59:29.641819 19219 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 02:59:29.641940 19219 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ranger-kms
I20260925 02:59:29.641988 19219 mini_ranger_kms.cc:227] Using postgres at 127.18.196.212:48915
2026-09-25 02:59:29,997 [I] DB FLAVOR :POSTGRES
2026-09-25 02:59:29,997 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:59:29,997 [I] Checking connection
2026-09-25 02:59:29,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:48915/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 02:59:30,245 [I] connection success
2026-09-25 02:59:30,246 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 02:59:30,246 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 02:59:30,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:48915/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 02:59:30,475 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 02:59:30,475 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:59:30,475 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 02:59:30,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.196.212:48915/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 02:59:30,716 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 02:59:31.174590 19219 mini_ranger_kms.cc:310] Created kms service
I20260925 02:59:31.401737 19219 mini_ranger_kms.cc:326] Created kudu user
I20260925 02:59:31.446954 19219 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 02:59:31 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305168, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.18.196.212@KRBTEST.COM
I20260925 02:59:31.838968 19219 mini_ranger_kms.cc:384] Added ranger policy
I20260925 02:59:31.839289 19219 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ranger-kms:/tmp/dist-test-taskM3GXH0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskM3GXH0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 02:59:31.839361 19219 mini_ranger_kms.cc:240] Using host: 127.18.196.212
I20260925 02:59:31.855505 19219 mini_ranger_kms.cc:276] Ranger KMS PID: 31555
I20260925 02:59:31.855628 19219 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.18.196.212:53869
Sep 25, 2026 2:59:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:59:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:59:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:59:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:59:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:59:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 02:59:33 dist-test-slave-8zvk krb5kdc[31161](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790305173, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.196.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:59:34.080807 19219 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.348s user 0.084s sys 0.282s
I20260925 02:59:34.081075 19219 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 02:59:34.081285 19219 mini_ranger_kms.cc:401] 127.18.196.212:53869/kms/v1/keys
Sep 25 02:59:34 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305173, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.196.212@KRBTEST.COM for HTTP/127.18.196.212@KRBTEST.COM
Sep 25 02:59:34 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305173, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.196.212@KRBTEST.COM for HTTP/127.18.196.212@KRBTEST.COM
Sep 25 02:59:37 dist-test-slave-8zvk krb5kdc[31161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 02:59:37.074083 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:39091
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:33579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:39091
--ranger_config_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.196.212:53869/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:59:37.423847 31607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:59:37.424162 31607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:59:37.424315 31607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:59:37.431227 31607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:59:37.431350 31607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:59:37.431457 31607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:59:37.431559 31607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:59:37.431638 31607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:59:37.441244 31607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:33579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/wal
--ranger_kms_url=127.18.196.212:53869/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:39091
--ranger_config_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:39091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:59:37.442440 31607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:59:37.443786 31607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:59:37.456138 31612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:37.456315 31615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:37.456476 31613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:59:37.457490 31607 server_base.cc:1034] running on GCE node
I20260925 02:59:37.458227 31607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:59:37.459568 31607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:59:37.460798 31607 hybrid_clock.cc:648] HybridClock initialized: now 1790305177460775 us; error 44 us; skew 500 ppm
I20260925 02:59:37.461109 31607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:59:37 dist-test-slave-8zvk krb5kdc[31161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305177, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:59:37.469249 31607 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 02:59:37.470541 31607 webserver.cc:460] Webserver started at http://127.18.196.254:46027/ using document root <none> and password file <none>
I20260925 02:59:37.471201 31607 fs_manager.cc:359] Metadata directory not provided
I20260925 02:59:37.471339 31607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:59:37.471625 31607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:59:37 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305177, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for HTTP/127.18.196.212@KRBTEST.COM
I20260925 02:59:37.526932 31607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "08903717dd064378b76fbb1fb6e9969d"
format_stamp: "Formatted at 2026-09-25 02:59:37 on dist-test-slave-8zvk"
server_key: "32cfd3821c7063d18fe748635b698f74"
server_key_iv: "25d320cb28be19fbeca4f0942c11e5db"
server_key_version: "kuduclusterkey@0"
I20260925 02:59:37.527980 31607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "08903717dd064378b76fbb1fb6e9969d"
format_stamp: "Formatted at 2026-09-25 02:59:37 on dist-test-slave-8zvk"
server_key: "32cfd3821c7063d18fe748635b698f74"
server_key_iv: "25d320cb28be19fbeca4f0942c11e5db"
server_key_version: "kuduclusterkey@0"
I20260925 02:59:37.534886 31607 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260925 02:59:37.549788 31622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:37.551196 31607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:59:37.551354 31607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "08903717dd064378b76fbb1fb6e9969d"
format_stamp: "Formatted at 2026-09-25 02:59:37 on dist-test-slave-8zvk"
server_key: "32cfd3821c7063d18fe748635b698f74"
server_key_iv: "25d320cb28be19fbeca4f0942c11e5db"
server_key_version: "kuduclusterkey@0"
I20260925 02:59:37.551555 31607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:59:37.575268 31607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:59:37.579051 31607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:59:37.579340 31607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:59:37.598385 31607 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:39091
I20260925 02:59:37.598389 31673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:39091 every 8 connection(s)
I20260925 02:59:37.599762 31607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:59:37.602970 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 31607
I20260925 02:59:37.603163 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:59:37.603559 31674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:59:37.604015 19219 external_mini_cluster.cc:1428] Setting key 18e5f9a8365a49fba5cd62497143a55e
I20260925 02:59:37.621755 31674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d: Bootstrap starting.
I20260925 02:59:37.624913 31674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d: Neither blocks nor log segments found. Creating new log.
I20260925 02:59:37.626060 31674 log.cc:824] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:59:37.628929 31674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d: No bootstrap required, opened a new log
Sep 25 02:59:37 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:59:37.637665 31674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08903717dd064378b76fbb1fb6e9969d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39091 } }
I20260925 02:59:37.638015 31674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:59:37.638141 31674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08903717dd064378b76fbb1fb6e9969d, State: Initialized, Role: FOLLOWER
I20260925 02:59:37.638711 31674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08903717dd064378b76fbb1fb6e9969d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39091 } }
I20260925 02:59:37.638969 31674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:59:37.639107 31674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:59:37.639279 31674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:59:37.641790 31674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08903717dd064378b76fbb1fb6e9969d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39091 } }
I20260925 02:59:37.642362 31674 leader_election.cc:302] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08903717dd064378b76fbb1fb6e9969d; no voters:
I20260925 02:59:37.643015 31674 leader_election.cc:288] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:59:37.643144 31679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:59:37.644011 31677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:37.606172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.18.196.254:39091)
0925 02:59:37.620881 (+ 14709us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:37.620912 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:37.620960 (+ 48us) server_negotiation.cc:407] Connection header received
0925 02:59:37.623388 (+ 2428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:37.623418 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:37.623894 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:37.624295 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:37.625812 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:37.627841 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:37.629463 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:37.630471 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:37.630657 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:37.635270 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:37.635280 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:37.635301 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:37.635345 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:37.639158 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:37.639989 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:37.640001 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:37.640008 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:37.640199 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:37.640617 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:37.640622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:37.640625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:37.641287 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:37.641497 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:37.642080 (+ 583us) server_negotiation.cc:299] Negotiation successful
0925 02:59:37.642470 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14480,"thread_start_us":15650,"threads_started":1}
I20260925 02:59:37.644624 31679 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [term 1 LEADER]: Becoming Leader. State: Replica: 08903717dd064378b76fbb1fb6e9969d, State: Running, Role: LEADER
I20260925 02:59:37.645040 31674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:59:37.645099 31679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08903717dd064378b76fbb1fb6e9969d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39091 } }
I20260925 02:59:37.647890 31681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08903717dd064378b76fbb1fb6e9969d. Latest consensus state: current_term: 1 leader_uuid: "08903717dd064378b76fbb1fb6e9969d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08903717dd064378b76fbb1fb6e9969d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39091 } } }
I20260925 02:59:37.648406 31681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [sys.catalog]: This master's current role is: LEADER
I20260925 02:59:37.647970 31680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08903717dd064378b76fbb1fb6e9969d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08903717dd064378b76fbb1fb6e9969d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39091 } } }
I20260925 02:59:37.648919 31680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [sys.catalog]: This master's current role is: LEADER
I20260925 02:59:37.649397 31687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:59:37.653911 31687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:59:37.661664 31687 catalog_manager.cc:1347] Generated new cluster ID: c108329078dd4b989df57f8b1068a616
I20260925 02:59:37.661820 31687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:59:37.667162 31674 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 02:59:37.680872 31687 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:59:37.682214 31687 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:59:37.693732 31687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d: Generated new TSK 0
I20260925 02:59:37.694583 31687 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
I20260925 02:59:38.435460 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.196.212:53869/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:39091
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:33579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:59:38 dist-test-slave-8zvk krb5kdc[31161](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790305178, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 02:59:38.883086 31719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:59:38.883512 31719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:59:38.883692 31719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:59:38.895191 31719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:59:38.895334 31719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:59:38.895536 31719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 02:59:38.911187 31719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:33579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.18.196.212:53869/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:39091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:59:38.912973 31719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:59:38.914672 31719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:59:38.930356 31729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:38.931107 31732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:38.932397 31730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:59:38.933260 31719 server_base.cc:1034] running on GCE node
I20260925 02:59:38.933935 31719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:59:38.935166 31719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:59:38.936419 31719 hybrid_clock.cc:648] HybridClock initialized: now 1790305178936363 us; error 50 us; skew 500 ppm
I20260925 02:59:38.936754 31719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:59:38 dist-test-slave-8zvk krb5kdc[31161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305178, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:59:38.940021 31719 init.cc:377] Logged in from keytab as kudu/127.18.196.193@KRBTEST.COM (short username kudu)
I20260925 02:59:38.948223 31719 webserver.cc:460] Webserver started at http://127.18.196.193:39695/ using document root <none> and password file <none>
I20260925 02:59:38.948983 31719 fs_manager.cc:359] Metadata directory not provided
I20260925 02:59:38.949131 31719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:59:38.949559 31719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:59:38 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305178, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for HTTP/127.18.196.212@KRBTEST.COM
I20260925 02:59:38.968694 31719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "30e09a0b8c9b405d9c89640992042b0e"
format_stamp: "Formatted at 2026-09-25 02:59:38 on dist-test-slave-8zvk"
server_key: "fafb9b624fba09d454cce3562516ce7c"
server_key_iv: "2644ba40f6051784d9e986fa4881c39f"
server_key_version: "kuduclusterkey@0"
I20260925 02:59:38.969575 31719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "30e09a0b8c9b405d9c89640992042b0e"
format_stamp: "Formatted at 2026-09-25 02:59:38 on dist-test-slave-8zvk"
server_key: "fafb9b624fba09d454cce3562516ce7c"
server_key_iv: "2644ba40f6051784d9e986fa4881c39f"
server_key_version: "kuduclusterkey@0"
I20260925 02:59:38.984938 31719 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.008s sys 0.008s
I20260925 02:59:39.001396 31739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:39.003039 31719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:59:39.003235 31719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "30e09a0b8c9b405d9c89640992042b0e"
format_stamp: "Formatted at 2026-09-25 02:59:38 on dist-test-slave-8zvk"
server_key: "fafb9b624fba09d454cce3562516ce7c"
server_key_iv: "2644ba40f6051784d9e986fa4881c39f"
server_key_version: "kuduclusterkey@0"
I20260925 02:59:39.003484 31719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:59:39.020695 31719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:59:39.025916 31719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:59:39.026298 31719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:59:39.027565 31719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:59:39.029196 31719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:59:39.029292 31719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:39.029456 31719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:59:39.029531 31719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:39.089195 31719 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:34409
I20260925 02:59:39.091202 31719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 02:59:39.089154 31851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:34409 every 8 connection(s)
I20260925 02:59:39.097507 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 31719
I20260925 02:59:39.097734 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 02:59:39.098575 19219 external_mini_cluster.cc:1428] Setting key d0d1b148659023fe7ee6c97c0f3ce456
Sep 25 02:59:39 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305178, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.194@KRBTEST.COM" created.
I20260925 02:59:39.159340 31855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:39.111883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39091 (local address 127.18.196.193:38095)
0925 02:59:39.133442 (+ 21559us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:39.133499 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:39.136122 (+ 2623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:39.138984 (+ 2862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:39.138998 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:39.139692 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:59:39.140555 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:39.140584 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:39.142249 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:39.142255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:39.144318 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:39.144334 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:39.145739 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:39.145747 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:39.146091 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:39.147032 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:59:39.147066 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:59:39.149964 (+ 2898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:59:39.154187 (+ 4223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:39.154197 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:39.154215 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:39.154866 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:39.155414 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:39.155419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:39.155423 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:39.155712 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:39.156520 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:59:39.156530 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:59:39.156991 (+ 461us) client_negotiation.cc:769] Sending connection context
0925 02:59:39.157320 (+ 329us) client_negotiation.cc:240] Negotiation successful
0925 02:59:39.157894 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21199,"thread_start_us":23481,"threads_started":1}
I20260925 02:59:39.160338 31854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:39.111236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:38095 (local address 127.18.196.254:39091)
0925 02:59:39.138333 (+ 27097us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:39.138339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:39.138361 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:59:39.138424 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:39.138429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:39.138506 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:39.138622 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:39.140833 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:39.142040 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:39.144528 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:39.145272 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:39.145409 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:39.150278 (+ 4869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:39.150285 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:39.150290 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:39.150332 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:39.154014 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:39.155113 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:39.155120 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:39.155123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:39.155290 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:39.155861 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:39.155867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:39.155871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:39.156274 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:39.156554 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:39.159711 (+ 3157us) server_negotiation.cc:299] Negotiation successful
0925 02:59:39.159946 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26957,"thread_start_us":27490,"threads_started":1}
I20260925 02:59:39.162390 31852 heartbeater.cc:347] Connected to a master server at 127.18.196.254:39091
I20260925 02:59:39.162743 31852 heartbeater.cc:464] Registering TS with master...
I20260925 02:59:39.163625 31852 heartbeater.cc:511] Master 127.18.196.254:39091 requested a full tablet report, sending...
I20260925 02:59:39.166119 31639 ts_manager.cc:194] Registered new tserver with Master: 30e09a0b8c9b405d9c89640992042b0e (127.18.196.193:34409)
I20260925 02:59:39.168183 31639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.193@KRBTEST.COM'} at 127.18.196.193:38095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
I20260925 02:59:39.199404 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.196.212:53869/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:39091
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:33579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:59:39 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305178, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for HTTP/127.18.196.212@KRBTEST.COM
Sep 25 02:59:39 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305178, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for HTTP/127.18.196.212@KRBTEST.COM
W20260925 02:59:39.633469 31858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:59:39.633961 31858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:59:39.634176 31858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:59:39.645013 31858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:59:39.645154 31858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:59:39.645386 31858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 02:59:39.660358 31858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:33579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.18.196.212:53869/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:39091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:59:39.662068 31858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:59:39.663870 31858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:59:39.681380 31866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:39.682173 31865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:39.689184 31869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:59:39.689659 31858 server_base.cc:1034] running on GCE node
I20260925 02:59:39.690594 31858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:59:39.695317 31858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:59:39.698755 31858 hybrid_clock.cc:648] HybridClock initialized: now 1790305179698644 us; error 82 us; skew 500 ppm
I20260925 02:59:39.699188 31858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:59:39 dist-test-slave-8zvk krb5kdc[31161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305179, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:59:39.703397 31858 init.cc:377] Logged in from keytab as kudu/127.18.196.194@KRBTEST.COM (short username kudu)
I20260925 02:59:39.705559 31858 webserver.cc:460] Webserver started at http://127.18.196.194:41305/ using document root <none> and password file <none>
I20260925 02:59:39.706502 31858 fs_manager.cc:359] Metadata directory not provided
I20260925 02:59:39.706677 31858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:59:39.707091 31858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:59:39 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305179, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for HTTP/127.18.196.212@KRBTEST.COM
I20260925 02:59:39.725786 31858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "9b484aa0c156475f8a2bdd6549940178"
format_stamp: "Formatted at 2026-09-25 02:59:39 on dist-test-slave-8zvk"
server_key: "1afdf4b0c0f01916a9ce8d5f6d9b5a9b"
server_key_iv: "18ef7b06d724fb1eea97fd76eef3ed3b"
server_key_version: "kuduclusterkey@0"
I20260925 02:59:39.726977 31858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "9b484aa0c156475f8a2bdd6549940178"
format_stamp: "Formatted at 2026-09-25 02:59:39 on dist-test-slave-8zvk"
server_key: "1afdf4b0c0f01916a9ce8d5f6d9b5a9b"
server_key_iv: "18ef7b06d724fb1eea97fd76eef3ed3b"
server_key_version: "kuduclusterkey@0"
I20260925 02:59:39.740049 31858 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.005s sys 0.004s
I20260925 02:59:39.773627 31875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:39.775621 31858 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.009s sys 0.000s
I20260925 02:59:39.775938 31858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "9b484aa0c156475f8a2bdd6549940178"
format_stamp: "Formatted at 2026-09-25 02:59:39 on dist-test-slave-8zvk"
server_key: "1afdf4b0c0f01916a9ce8d5f6d9b5a9b"
server_key_iv: "18ef7b06d724fb1eea97fd76eef3ed3b"
server_key_version: "kuduclusterkey@0"
I20260925 02:59:39.776285 31858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:59:39.815788 31858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:59:39.821112 31858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:59:39.821529 31858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:59:39.828934 31858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:59:39.837023 31858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:59:39.837183 31858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260925 02:59:39.837363 31858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:59:39.837476 31858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:39.912087 31858 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:40213
I20260925 02:59:39.912106 31994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:40213 every 8 connection(s)
I20260925 02:59:39.913543 31858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 02:59:39.917732 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 31858
I20260925 02:59:39.917908 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 02:59:39.918475 19219 external_mini_cluster.cc:1428] Setting key 30d7de9aeada333c83e4a77547b170b1
Sep 25 02:59:39 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305179, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:59:39.944351 31998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:39.918171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:40757 (local address 127.18.196.254:39091)
0925 02:59:39.923292 (+ 5121us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:39.923298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:39.925446 (+ 2148us) server_negotiation.cc:407] Connection header received
0925 02:59:39.926636 (+ 1190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:39.926641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:39.926719 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:39.926851 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:39.928867 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:39.929932 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:39.932251 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:39.932862 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:39.932952 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:39.937245 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:39.937254 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:39.937261 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:39.937309 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:39.940704 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:39.941574 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:39.941579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:39.941583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:39.941752 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:39.942296 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:39.942301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:39.942304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:39.942648 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:39.942795 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:39.943734 (+ 939us) server_negotiation.cc:299] Negotiation successful
0925 02:59:39.943991 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4977,"thread_start_us":4852,"threads_started":1}
I20260925 02:59:39.946869 31997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:39.917508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39091 (local address 127.18.196.194:40757)
0925 02:59:39.923788 (+ 6280us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:39.923834 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:39.926226 (+ 2392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:39.927258 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:39.927270 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:39.927891 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:59:39.928661 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:39.928691 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:39.930116 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:39.930121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:39.932096 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:39.932119 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:39.933250 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:39.933258 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:39.933531 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:39.934295 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:59:39.934323 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:59:39.937002 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:59:39.940871 (+ 3869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:39.940880 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:39.940896 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:39.941427 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:39.941904 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:39.941909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:39.941912 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:39.942163 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:39.942922 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:59:39.942929 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:59:39.943313 (+ 384us) client_negotiation.cc:769] Sending connection context
0925 02:59:39.944925 (+ 1612us) client_negotiation.cc:240] Negotiation successful
0925 02:59:39.945413 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5978,"thread_start_us":6871,"threads_started":1}
I20260925 02:59:39.948690 31995 heartbeater.cc:347] Connected to a master server at 127.18.196.254:39091
I20260925 02:59:39.949009 31995 heartbeater.cc:464] Registering TS with master...
I20260925 02:59:39.949838 31995 heartbeater.cc:511] Master 127.18.196.254:39091 requested a full tablet report, sending...
I20260925 02:59:39.951526 31639 ts_manager.cc:194] Registered new tserver with Master: 9b484aa0c156475f8a2bdd6549940178 (127.18.196.194:40213)
WARNING: no policy specified for kudu/127.18.196.195@KRBTEST.COM; defaulting to no policy
I20260925 02:59:39.952591 31639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.194@KRBTEST.COM'} at 127.18.196.194:40757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
I20260925 02:59:39.981961 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.196.212:53869/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:39091
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:33579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:59:40.171780 31852 heartbeater.cc:503] Master 127.18.196.254:39091 was elected leader, sending a full tablet report...
W20260925 02:59:40.299611 32001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:59:40.299906 32001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:59:40.300066 32001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:59:40.308267 32001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:59:40.308383 32001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:59:40.308533 32001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 02:59:40.317466 32001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:33579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.18.196.212:53869/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:39091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:59:40.318540 32001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:59:40.319880 32001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:59:40.331991 32006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:40.332069 32007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:40.332031 32009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:59:40.333150 32001 server_base.cc:1034] running on GCE node
I20260925 02:59:40.333715 32001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:59:40.334514 32001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:59:40.335698 32001 hybrid_clock.cc:648] HybridClock initialized: now 1790305180335656 us; error 34 us; skew 500 ppm
I20260925 02:59:40.335994 32001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:59:40 dist-test-slave-8zvk krb5kdc[31161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305180, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:59:40.338999 32001 init.cc:377] Logged in from keytab as kudu/127.18.196.195@KRBTEST.COM (short username kudu)
I20260925 02:59:40.340313 32001 webserver.cc:460] Webserver started at http://127.18.196.195:42573/ using document root <none> and password file <none>
I20260925 02:59:40.340929 32001 fs_manager.cc:359] Metadata directory not provided
I20260925 02:59:40.341060 32001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:59:40.341333 32001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:59:40 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305180, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for HTTP/127.18.196.212@KRBTEST.COM
I20260925 02:59:40.356120 32001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "e7634175f3874cddaac517c7a44b12e4"
format_stamp: "Formatted at 2026-09-25 02:59:40 on dist-test-slave-8zvk"
server_key: "cc8e286fea07fc301544d8dd773a08e2"
server_key_iv: "5b7334a9012b1fe695d84db45a2081d5"
server_key_version: "kuduclusterkey@0"
I20260925 02:59:40.356901 32001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "e7634175f3874cddaac517c7a44b12e4"
format_stamp: "Formatted at 2026-09-25 02:59:40 on dist-test-slave-8zvk"
server_key: "cc8e286fea07fc301544d8dd773a08e2"
server_key_iv: "5b7334a9012b1fe695d84db45a2081d5"
server_key_version: "kuduclusterkey@0"
I20260925 02:59:40.361704 32001 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:59:40.376637 32016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:40.378005 32001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:59:40.378175 32001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "e7634175f3874cddaac517c7a44b12e4"
format_stamp: "Formatted at 2026-09-25 02:59:40 on dist-test-slave-8zvk"
server_key: "cc8e286fea07fc301544d8dd773a08e2"
server_key_iv: "5b7334a9012b1fe695d84db45a2081d5"
server_key_version: "kuduclusterkey@0"
I20260925 02:59:40.378443 32001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:59:40.399405 32001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:59:40.403445 32001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:59:40.403751 32001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:59:40.404663 32001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:59:40.405957 32001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:59:40.406060 32001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:40.406209 32001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:59:40.406281 32001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:40.438395 32001 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:33053
I20260925 02:59:40.438476 32128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:33053 every 8 connection(s)
I20260925 02:59:40.439663 32001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 02:59:40.441959 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 32001
I20260925 02:59:40.442150 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 02:59:40.442619 19219 external_mini_cluster.cc:1428] Setting key e6a40245c02dd61a3f6ef2f75d1022c8
Sep 25 02:59:40 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305180, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:59:40.458998 31998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:40.442203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:53699 (local address 127.18.196.254:39091)
0925 02:59:40.442405 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:40.442411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:40.443807 (+ 1396us) server_negotiation.cc:407] Connection header received
0925 02:59:40.445934 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:40.445939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:40.446036 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:40.446117 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:40.447510 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:40.448217 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:40.449636 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:40.450067 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:40.450169 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:40.453167 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:40.453171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:40.453174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:40.453203 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:40.455831 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:40.456713 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:40.456718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:40.456724 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:40.456883 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:40.457347 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:40.457351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:40.457353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:40.457670 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:40.457820 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:40.458563 (+ 743us) server_negotiation.cc:299] Negotiation successful
0925 02:59:40.458726 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 02:59:40.459856 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:40.442698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39091 (local address 127.18.196.195:53699)
0925 02:59:40.443642 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:40.443693 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:40.445678 (+ 1985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:40.446323 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:40.446330 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:40.446766 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:59:40.447371 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:40.447388 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:40.448341 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:40.448344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:40.449506 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:40.449525 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:40.450253 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:40.450257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:40.450442 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:40.451062 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:59:40.451081 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:59:40.452982 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:59:40.455962 (+ 2980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:40.455968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:40.455978 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:40.456560 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:40.456986 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:40.456989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:40.456991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:40.457216 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:40.457805 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:59:40.457813 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:59:40.458161 (+ 348us) client_negotiation.cc:769] Sending connection context
0925 02:59:40.458462 (+ 301us) client_negotiation.cc:240] Negotiation successful
0925 02:59:40.458797 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":362,"threads_started":1}
I20260925 02:59:40.461099 32129 heartbeater.cc:347] Connected to a master server at 127.18.196.254:39091
I20260925 02:59:40.461407 32129 heartbeater.cc:464] Registering TS with master...
I20260925 02:59:40.462076 32129 heartbeater.cc:511] Master 127.18.196.254:39091 requested a full tablet report, sending...
I20260925 02:59:40.463407 31639 ts_manager.cc:194] Registered new tserver with Master: e7634175f3874cddaac517c7a44b12e4 (127.18.196.195:33053)
I20260925 02:59:40.464115 31639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.195@KRBTEST.COM'} at 127.18.196.195:53699
I20260925 02:59:40.470275 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:59:40.472010 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 31719
Sep 25 02:59:40 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.193@KRBTEST.COM
I20260925 02:59:40.487187 32133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:40.472904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48660 (local address 127.18.196.193:34409)
0925 02:59:40.473541 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:40.473548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:40.473569 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:59:40.473626 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:40.473632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:40.473801 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:40.473917 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:40.475261 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:40.476198 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:40.477638 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:40.478148 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:40.478236 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:40.482361 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:40.482371 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:40.482382 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:40.482414 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:40.484967 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:40.485550 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:40.485555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:40.485557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:40.485679 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:40.485960 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:40.485963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:40.485964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:40.486256 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:40.486472 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:40.486672 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 02:59:40.486921 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":336,"threads_started":1}
I20260925 02:59:40.488300 31847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:59:40.570084 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 31719
I20260925 02:59:40.586136 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 31858
Sep 25 02:59:40 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.194@KRBTEST.COM
I20260925 02:59:40.602636 32135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:40.587084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46356 (local address 127.18.196.194:40213)
0925 02:59:40.587591 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:40.587597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:40.587621 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:59:40.587679 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:40.587685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:40.587886 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:40.588009 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:40.589200 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:40.590156 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:40.591706 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:40.592489 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:40.592618 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:40.596888 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:40.596899 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:40.596914 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:40.596957 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:40.599789 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:40.600615 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:40.600621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:40.600623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:40.600777 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:40.601151 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:40.601154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:40.601156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:40.601481 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:40.601698 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:40.602055 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 02:59:40.602266 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":258,"threads_started":1}
I20260925 02:59:40.604094 31990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:59:40.683988 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 31858
I20260925 02:59:40.700508 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 32001
Sep 25 02:59:40 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.195@KRBTEST.COM
I20260925 02:59:40.717123 32137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:40.701526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34478 (local address 127.18.196.195:33053)
0925 02:59:40.702207 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:40.702214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:40.702250 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:59:40.702308 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:40.702315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:40.702523 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:40.702646 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:40.703962 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:40.704958 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:40.706217 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:40.706768 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:40.706949 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:40.711340 (+ 4391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:40.711351 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:40.711365 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:40.711396 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:40.714218 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:40.714975 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:40.714982 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:40.714984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:40.715122 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:40.715519 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:40.715528 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:40.715530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:40.715851 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:40.716202 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:40.716521 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 02:59:40.716777 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":364,"threads_started":1}
I20260925 02:59:40.718405 32124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:59:40.800565 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 32001
I20260925 02:59:40.816695 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 31607
I20260925 02:59:40.818034 31669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:59:40.924121 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 31607
I20260925 02:59:40.945936 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:39091
--webserver_interface=127.18.196.254
--webserver_port=46027
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:33579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:39091
--ranger_config_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.196.212:53869/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.196.212:53869/kms with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:59:41.259516 32140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:59:41.259809 32140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:59:41.259958 32140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:59:41.266434 32140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:59:41.266571 32140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:59:41.266677 32140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:59:41.266768 32140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:59:41.266870 32140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:59:41.275776 32140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:33579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.196.212:53869/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:39091
--ranger_config_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:39091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=46027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:59:41.276882 32140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:59:41.278072 32140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:59:41.289018 32145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:41.288970 32146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:41.289065 32148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:59:41.290046 32140 server_base.cc:1034] running on GCE node
I20260925 02:59:41.290737 32140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:59:41.292140 32140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:59:41.293370 32140 hybrid_clock.cc:648] HybridClock initialized: now 1790305181293298 us; error 89 us; skew 500 ppm
I20260925 02:59:41.293671 32140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:59:41 dist-test-slave-8zvk krb5kdc[31161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305181, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:59:41.296687 32140 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 02:59:41.298002 32140 webserver.cc:460] Webserver started at http://127.18.196.254:46027/ using document root <none> and password file <none>
I20260925 02:59:41.298586 32140 fs_manager.cc:359] Metadata directory not provided
I20260925 02:59:41.298691 32140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:59:41 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305181, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for HTTP/127.18.196.212@KRBTEST.COM
I20260925 02:59:41.410712 32140 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 02:59:41.413715 32156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:41.415081 32140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 02:59:41.415264 32140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "08903717dd064378b76fbb1fb6e9969d"
format_stamp: "Formatted at 2026-09-25 02:59:37 on dist-test-slave-8zvk"
server_key: "32cfd3821c7063d18fe748635b698f74"
server_key_iv: "25d320cb28be19fbeca4f0942c11e5db"
server_key_version: "kuduclusterkey@0"
I20260925 02:59:41.416214 32140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:59:41.431605 32140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:59:41.435073 32140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:59:41.435372 32140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:59:41.453359 32140 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:39091
I20260925 02:59:41.453377 32207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:39091 every 8 connection(s)
I20260925 02:59:41.454784 32140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:59:41.458748 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 32140
I20260925 02:59:41.459601 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.196.212:53869/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:34409
--local_ip_for_outbound_sockets=127.18.196.193
--tserver_master_addrs=127.18.196.254:39091
--webserver_port=39695
--webserver_interface=127.18.196.193
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:33579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.196.212:53869/kms with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:59:41.461452 32208 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:59:41.464612 32208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d: Bootstrap starting.
I20260925 02:59:41.483103 32208 log.cc:824] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:59:41.492122 32208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:59:41.492704 32208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d: Bootstrap complete.
I20260925 02:59:41.501807 32208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08903717dd064378b76fbb1fb6e9969d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39091 } }
I20260925 02:59:41.502669 32208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08903717dd064378b76fbb1fb6e9969d, State: Initialized, Role: FOLLOWER
I20260925 02:59:41.503322 32208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08903717dd064378b76fbb1fb6e9969d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39091 } }
I20260925 02:59:41.503562 32208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:59:41.503683 32208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:59:41.503865 32208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:59:41.507648 32208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08903717dd064378b76fbb1fb6e9969d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39091 } }
I20260925 02:59:41.508139 32208 leader_election.cc:302] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08903717dd064378b76fbb1fb6e9969d; no voters:
I20260925 02:59:41.508775 32208 leader_election.cc:288] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:59:41.508916 32212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:59:41.510114 32212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [term 2 LEADER]: Becoming Leader. State: Replica: 08903717dd064378b76fbb1fb6e9969d, State: Running, Role: LEADER
I20260925 02:59:41.510746 32212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08903717dd064378b76fbb1fb6e9969d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39091 } }
I20260925 02:59:41.511013 32208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:59:41.513425 32214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08903717dd064378b76fbb1fb6e9969d. Latest consensus state: current_term: 2 leader_uuid: "08903717dd064378b76fbb1fb6e9969d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08903717dd064378b76fbb1fb6e9969d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39091 } } }
I20260925 02:59:41.513470 32213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "08903717dd064378b76fbb1fb6e9969d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08903717dd064378b76fbb1fb6e9969d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39091 } } }
I20260925 02:59:41.513805 32214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [sys.catalog]: This master's current role is: LEADER
I20260925 02:59:41.513846 32213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d [sys.catalog]: This master's current role is: LEADER
I20260925 02:59:41.526129 32219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:59:41.535260 32219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:59:41.536913 32219 catalog_manager.cc:1259] Loaded cluster ID: c108329078dd4b989df57f8b1068a616
I20260925 02:59:41.537061 32219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:59:41.540794 32219 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:59:41.542340 32219 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d: Loaded TSK: 0
I20260925 02:59:41.543797 32219 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:59:41.546749 32208 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260925 02:59:41.891538 32210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:59:41.892014 32210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:59:41.892228 32210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:59:41.904168 32210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:59:41.904420 32210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:59:41.904670 32210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 02:59:41.920156 32210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:33579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.196.212:53869/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:34409
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=39695
--tserver_master_addrs=127.18.196.254:39091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:59:41.921849 32210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:59:41.923436 32210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:59:41.940096 32245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:41.948956 32246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:41.949671 32249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:59:41.950999 32210 server_base.cc:1034] running on GCE node
I20260925 02:59:41.951741 32210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:59:41.955168 32210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:59:41.956492 32210 hybrid_clock.cc:648] HybridClock initialized: now 1790305181956423 us; error 56 us; skew 500 ppm
I20260925 02:59:41.956862 32210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:59:41 dist-test-slave-8zvk krb5kdc[31161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305181, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:59:41.963948 32210 init.cc:377] Logged in from keytab as kudu/127.18.196.193@KRBTEST.COM (short username kudu)
I20260925 02:59:41.998595 32210 webserver.cc:460] Webserver started at http://127.18.196.193:39695/ using document root <none> and password file <none>
I20260925 02:59:41.999782 32210 fs_manager.cc:359] Metadata directory not provided
I20260925 02:59:41.999954 32210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:59:42 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305181, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for HTTP/127.18.196.212@KRBTEST.COM
I20260925 02:59:42.045838 32210 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.001s
I20260925 02:59:42.056865 32257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:42.058724 32210 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.009s sys 0.000s
I20260925 02:59:42.058988 32210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "30e09a0b8c9b405d9c89640992042b0e"
format_stamp: "Formatted at 2026-09-25 02:59:38 on dist-test-slave-8zvk"
server_key: "fafb9b624fba09d454cce3562516ce7c"
server_key_iv: "2644ba40f6051784d9e986fa4881c39f"
server_key_version: "kuduclusterkey@0"
I20260925 02:59:42.060542 32210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:59:42.111430 32210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:59:42.116209 32210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:59:42.116636 32210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:59:42.123960 32210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:59:42.125765 32210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:59:42.125890 32210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:42.126026 32210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:59:42.126128 32210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:42.297641 32210 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:34409
I20260925 02:59:42.297712 32369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:34409 every 8 connection(s)
I20260925 02:59:42.299830 32210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 02:59:42.304838 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 32210
I20260925 02:59:42.305709 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.196.212:53869/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:40213
--local_ip_for_outbound_sockets=127.18.196.194
--tserver_master_addrs=127.18.196.254:39091
--webserver_port=41305
--webserver_interface=127.18.196.194
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:33579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.196.212:53869/kms with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:59:42 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305181, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:59:42.352923 32372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:42.303887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39091 (local address 127.18.196.193:58167)
0925 02:59:42.325435 (+ 21548us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:42.325490 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:42.328235 (+ 2745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:42.332139 (+ 3904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:42.332150 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:42.332892 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:59:42.333732 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:42.333759 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:42.336419 (+ 2660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:42.336427 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:42.338570 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:42.338589 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:42.340562 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:42.340570 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:42.340872 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:42.341659 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:59:42.341695 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:59:42.344195 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:59:42.348415 (+ 4220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:42.348423 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:42.348436 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:42.348950 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:42.349462 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:42.349467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:42.349470 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:42.349664 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:42.350522 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:59:42.350530 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:59:42.350910 (+ 380us) client_negotiation.cc:769] Sending connection context
0925 02:59:42.351170 (+ 260us) client_negotiation.cc:240] Negotiation successful
0925 02:59:42.351661 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21178,"thread_start_us":24650,"threads_started":1}
I20260925 02:59:42.355928 32373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:42.309426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:58167 (local address 127.18.196.254:39091)
0925 02:59:42.329909 (+ 20483us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:42.329928 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:42.329978 (+ 50us) server_negotiation.cc:407] Connection header received
0925 02:59:42.330457 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:42.330476 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:42.330976 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:42.331403 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:42.333986 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:42.336173 (+ 2187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:42.338778 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:42.339915 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:42.340130 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:42.344509 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:42.344518 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:42.344536 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:42.344581 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:42.348251 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:42.349135 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:42.349143 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:42.349150 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:42.349325 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:42.349786 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:42.349791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:42.349794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:42.350369 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:42.354280 (+ 3911us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:42.354519 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 02:59:42.354896 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20201,"thread_start_us":23967,"threads_started":1}
I20260925 02:59:42.359521 32370 heartbeater.cc:347] Connected to a master server at 127.18.196.254:39091
I20260925 02:59:42.359853 32370 heartbeater.cc:464] Registering TS with master...
I20260925 02:59:42.360567 32370 heartbeater.cc:511] Master 127.18.196.254:39091 requested a full tablet report, sending...
W20260925 02:59:42.363487 32370 heartbeater.cc:644] Failed to heartbeat to 127.18.196.254:39091 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 02:59:42.363731 32370 heartbeater.cc:464] Registering TS with master...
I20260925 02:59:42.364025 32370 heartbeater.cc:511] Master 127.18.196.254:39091 requested a full tablet report, sending...
I20260925 02:59:42.365120 32370 heartbeater.cc:464] Registering TS with master...
I20260925 02:59:42.365406 32370 heartbeater.cc:511] Master 127.18.196.254:39091 requested a full tablet report, sending...
W20260925 02:59:42.366406 32370 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:59:42.818194 32374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:59:42.818599 32374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:59:42.818773 32374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:59:42.828085 32374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:59:42.828182 32374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:59:42.828359 32374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
Sep 25 02:59:42 dist-test-slave-8zvk krb5kdc[31161](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790305182, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:59:42.839689 32374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:33579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.196.212:53869/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:40213
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=41305
--tserver_master_addrs=127.18.196.254:39091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:59:42.841070 32374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:59:42.842474 32374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:59:42.869796 32392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:42.870522 32391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:42.871807 32395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:59:42.872637 32374 server_base.cc:1034] running on GCE node
I20260925 02:59:42.873358 32374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:59:42.874516 32374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:59:42.875814 32374 hybrid_clock.cc:648] HybridClock initialized: now 1790305182875753 us; error 43 us; skew 500 ppm
I20260925 02:59:42.876173 32374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:59:42 dist-test-slave-8zvk krb5kdc[31161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305182, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:59:42.879520 32374 init.cc:377] Logged in from keytab as kudu/127.18.196.194@KRBTEST.COM (short username kudu)
I20260925 02:59:42.881211 32374 webserver.cc:460] Webserver started at http://127.18.196.194:41305/ using document root <none> and password file <none>
I20260925 02:59:42.881901 32374 fs_manager.cc:359] Metadata directory not provided
I20260925 02:59:42.882041 32374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:59:42 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305182, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for HTTP/127.18.196.212@KRBTEST.COM
I20260925 02:59:42.909971 32374 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 02:59:42.913041 32403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:42.915069 32374 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 02:59:42.915269 32374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "9b484aa0c156475f8a2bdd6549940178"
format_stamp: "Formatted at 2026-09-25 02:59:39 on dist-test-slave-8zvk"
server_key: "1afdf4b0c0f01916a9ce8d5f6d9b5a9b"
server_key_iv: "18ef7b06d724fb1eea97fd76eef3ed3b"
server_key_version: "kuduclusterkey@0"
I20260925 02:59:42.916288 32374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:59:42.936611 32374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:59:42.940608 32374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:59:42.940873 32374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:59:42.941931 32374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:59:42.943322 32374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:59:42.943439 32374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:42.943570 32374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:59:42.943655 32374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:43.004271 32374 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:40213
I20260925 02:59:43.004339 32516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:40213 every 8 connection(s)
I20260925 02:59:43.006127 32374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 02:59:43.015681 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 32374
I20260925 02:59:43.016427 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.196.212:53869/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:33053
--local_ip_for_outbound_sockets=127.18.196.195
--tserver_master_addrs=127.18.196.254:39091
--webserver_port=42573
--webserver_interface=127.18.196.195
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:33579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.196.212:53869/kms with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:59:43 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305182, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:59:43.071686 32520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:43.013192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:46117 (local address 127.18.196.254:39091)
0925 02:59:43.048025 (+ 34833us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:43.048032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:43.050933 (+ 2901us) server_negotiation.cc:407] Connection header received
0925 02:59:43.052579 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:43.052598 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:43.052689 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:43.052768 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:43.055913 (+ 3145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:43.057103 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:43.059463 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:43.060100 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:43.060195 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:43.064677 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:43.064683 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:43.064688 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:43.064727 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:43.068410 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:43.069315 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:43.069321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:43.069324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:43.069470 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:43.069993 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:43.069997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:43.070000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:43.070260 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:43.070413 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:43.071107 (+ 694us) server_negotiation.cc:299] Negotiation successful
0925 02:59:43.071311 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34680,"thread_start_us":34847,"threads_started":1}
I20260925 02:59:43.072654 32519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:43.012546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39091 (local address 127.18.196.194:46117)
0925 02:59:43.048687 (+ 36141us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:43.048741 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:43.052256 (+ 3515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:43.053340 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:43.053352 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:43.053961 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:59:43.054727 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:43.054752 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:43.057317 (+ 2565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:43.057322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:43.059305 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:43.059320 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:43.060537 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:43.060543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:43.060815 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:43.061646 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:59:43.061675 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:59:43.064403 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:59:43.068574 (+ 4171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:43.068605 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:43.068621 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:43.069178 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:43.069644 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:43.069651 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:43.069656 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:43.069829 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:43.070392 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:59:43.070400 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:59:43.070697 (+ 297us) client_negotiation.cc:769] Sending connection context
0925 02:59:43.071007 (+ 310us) client_negotiation.cc:240] Negotiation successful
0925 02:59:43.071504 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35787,"thread_start_us":38267,"threads_started":1}
I20260925 02:59:43.074112 32517 heartbeater.cc:347] Connected to a master server at 127.18.196.254:39091
I20260925 02:59:43.074477 32517 heartbeater.cc:464] Registering TS with master...
I20260925 02:59:43.075350 32517 heartbeater.cc:511] Master 127.18.196.254:39091 requested a full tablet report, sending...
I20260925 02:59:43.077421 32173 ts_manager.cc:194] Registered new tserver with Master: 9b484aa0c156475f8a2bdd6549940178 (127.18.196.194:40213)
I20260925 02:59:43.079341 32173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.194@KRBTEST.COM'} at 127.18.196.194:46117
Sep 25 02:59:43 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305182, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for HTTP/127.18.196.212@KRBTEST.COM
I20260925 02:59:43.366564 32370 heartbeater.cc:464] Registering TS with master...
I20260925 02:59:43.367075 32370 heartbeater.cc:511] Master 127.18.196.254:39091 requested a full tablet report, sending...
I20260925 02:59:43.368417 32173 ts_manager.cc:194] Registered new tserver with Master: 30e09a0b8c9b405d9c89640992042b0e (127.18.196.193:34409)
I20260925 02:59:43.369587 32173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.193@KRBTEST.COM'} at 127.18.196.193:58167
W20260925 02:59:43.479839 32521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:59:43.480199 32521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:59:43.480407 32521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:59:43.491874 32521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:59:43.492012 32521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:59:43.492249 32521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 02:59:43.507512 32521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:33579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.196.212:53869/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:33053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=42573
--tserver_master_addrs=127.18.196.254:39091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:59:43.509176 32521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:59:43.510879 32521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:59:43.528077 32526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:43.529531 32527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:43.540300 32529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:59:43.541164 32521 server_base.cc:1034] running on GCE node
I20260925 02:59:43.542009 32521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:59:43.543327 32521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:59:43.544642 32521 hybrid_clock.cc:648] HybridClock initialized: now 1790305183544576 us; error 53 us; skew 500 ppm
I20260925 02:59:43.545048 32521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:59:43 dist-test-slave-8zvk krb5kdc[31161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305183, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:59:43.549094 32521 init.cc:377] Logged in from keytab as kudu/127.18.196.195@KRBTEST.COM (short username kudu)
I20260925 02:59:43.554505 32521 webserver.cc:460] Webserver started at http://127.18.196.195:42573/ using document root <none> and password file <none>
I20260925 02:59:43.555476 32521 fs_manager.cc:359] Metadata directory not provided
I20260925 02:59:43.555696 32521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:59:43 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305183, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for HTTP/127.18.196.212@KRBTEST.COM
Sep 25 02:59:43 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305182, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for HTTP/127.18.196.212@KRBTEST.COM
I20260925 02:59:43.596836 32521 fs_manager.cc:709] Time spent opening directory manager: real 0.019s user 0.006s sys 0.007s
I20260925 02:59:43.606886 32538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:43.608728 32521 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.005s sys 0.004s
I20260925 02:59:43.609004 32521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "e7634175f3874cddaac517c7a44b12e4"
format_stamp: "Formatted at 2026-09-25 02:59:40 on dist-test-slave-8zvk"
server_key: "cc8e286fea07fc301544d8dd773a08e2"
server_key_iv: "5b7334a9012b1fe695d84db45a2081d5"
server_key_version: "kuduclusterkey@0"
I20260925 02:59:43.610597 32521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:59:43.662045 32521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:59:43.667654 32521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:59:43.668074 32521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:59:43.672452 32521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:59:43.683686 32521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:59:43.683825 32521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:43.683947 32521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:59:43.684031 32521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:43.812457 32521 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:33053
I20260925 02:59:43.812429 32650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:33053 every 8 connection(s)
I20260925 02:59:43.813974 32521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 02:59:43.816090 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 32521
Sep 25 02:59:43 dist-test-slave-8zvk krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305183, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:59:43.865888 32654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:43.817387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57894 (local address 127.18.196.254:39091)
0925 02:59:43.836060 (+ 18673us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:43.836070 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:43.844552 (+ 8482us) server_negotiation.cc:407] Connection header received
0925 02:59:43.844809 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:43.844815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:43.844916 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:43.845024 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:43.848659 (+ 3635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:43.849805 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:43.852918 (+ 3113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:43.853540 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:43.853888 (+ 348us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:43.857260 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:43.857267 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:43.857272 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:43.857334 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:43.860926 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:43.861847 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:43.861855 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:43.861863 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:43.861982 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:43.862974 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:43.862981 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:43.862987 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:43.863415 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:43.863731 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:43.865151 (+ 1420us) server_negotiation.cc:299] Negotiation successful
0925 02:59:43.865464 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18482,"thread_start_us":18683,"threads_started":1}
I20260925 02:59:43.870491 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:43.836290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:48777 (local address 127.18.196.254:39091)
0925 02:59:43.843654 (+ 7364us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:43.843661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:43.843684 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:59:43.846018 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:43.846023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:43.846108 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:43.846197 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:43.851171 (+ 4974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:43.852750 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:43.856751 (+ 4001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:43.857437 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:43.857537 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:43.862466 (+ 4929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:43.862471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:43.862474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:43.862506 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:43.865919 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:43.867753 (+ 1834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:43.867758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:43.867761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:43.867909 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:43.868454 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:43.868460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:43.868466 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:43.868917 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:43.869022 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:43.869899 (+ 877us) server_negotiation.cc:299] Negotiation successful
0925 02:59:43.870116 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7238,"thread_start_us":7410,"threads_started":1}
I20260925 02:59:43.871809 32655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:43.825476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39091 (local address 127.18.196.195:48777)
0925 02:59:43.840010 (+ 14534us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:43.840102 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:43.845645 (+ 5543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:43.849096 (+ 3451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:43.849110 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:43.849762 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:59:43.850855 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:43.850885 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:43.854343 (+ 3458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:43.854348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:43.856572 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:43.856589 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:43.857902 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:43.857912 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:43.858260 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:43.858992 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:59:43.859027 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:59:43.862174 (+ 3147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:59:43.866098 (+ 3924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:43.866109 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:43.866492 (+ 383us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:43.867396 (+ 904us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:43.868061 (+ 665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:43.868065 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:43.868069 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:43.868313 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:43.869097 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:59:43.869103 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:59:43.869536 (+ 433us) client_negotiation.cc:769] Sending connection context
0925 02:59:43.869808 (+ 272us) client_negotiation.cc:240] Negotiation successful
0925 02:59:43.870365 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14094,"thread_start_us":17230,"threads_started":1}
I20260925 02:59:43.873257 32651 heartbeater.cc:347] Connected to a master server at 127.18.196.254:39091
I20260925 02:59:43.873615 32651 heartbeater.cc:464] Registering TS with master...
I20260925 02:59:43.874475 32651 heartbeater.cc:511] Master 127.18.196.254:39091 requested a full tablet report, sending...
I20260925 02:59:43.876144 32173 ts_manager.cc:194] Registered new tserver with Master: e7634175f3874cddaac517c7a44b12e4 (127.18.196.195:33053)
I20260925 02:59:43.876952 32173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.195@KRBTEST.COM'} at 127.18.196.195:48777
I20260925 02:59:43.880911 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:59:43.897168 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:43.883914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57902 (local address 127.18.196.254:39091)
0925 02:59:43.884063 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:43.884067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:43.884537 (+ 470us) server_negotiation.cc:407] Connection header received
0925 02:59:43.884746 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:43.884752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:43.884818 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:43.884918 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:43.886373 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:43.887139 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:43.888522 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:43.888957 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:43.889087 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:43.891706 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:43.891711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:43.891714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:43.891745 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:43.894319 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:43.895116 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:43.895122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:43.895126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:43.895264 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:43.895679 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:43.895684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:43.895689 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:43.895977 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:43.896231 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:43.896598 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 02:59:43.896834 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:59:43.902426 32173 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:59:43.913226 32173 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:59:43.957134 32674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:43.929685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:34409 (local address 127.0.0.1:52304)
0925 02:59:43.945988 (+ 16303us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:43.946013 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:43.946781 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:43.947883 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:43.947891 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:43.947936 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:59:43.948360 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:43.948373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:43.950388 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:43.951389 (+ 1001us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:43.953883 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:43.953898 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:43.956085 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:43.956091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:43.956285 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:43.956352 (+ 67us) client_negotiation.cc:769] Sending connection context
0925 02:59:43.956494 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 02:59:43.956677 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16138,"mutex_wait_us":27,"thread_start_us":19199,"threads_started":1}
I20260925 02:59:43.959460 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:43.930065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52304 (local address 127.18.196.193:34409)
0925 02:59:43.943953 (+ 13888us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:43.943963 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:43.946141 (+ 2178us) server_negotiation.cc:407] Connection header received
0925 02:59:43.947211 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:43.947219 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:43.947453 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:43.947630 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:59:43.948562 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:43.950187 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:43.954059 (+ 3872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:43.955539 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:43.955713 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:43.955896 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:43.958725 (+ 2829us) server_negotiation.cc:299] Negotiation successful
0925 02:59:43.958965 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13704,"thread_start_us":15489,"threads_started":1}
I20260925 02:59:43.962707 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:43.942776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:33053 (local address 127.0.0.1:37700)
0925 02:59:43.954226 (+ 11450us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:43.954239 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:43.954372 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:43.955330 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:43.955335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:43.955358 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:59:43.955674 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:43.955685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:43.957630 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:43.957635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:43.959736 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:43.959751 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:43.961933 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:43.961939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:43.962134 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:43.962173 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 02:59:43.962267 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 02:59:43.962400 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11354,"thread_start_us":11199,"threads_started":1}
I20260925 02:59:43.963502 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:43.929851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46410 (local address 127.18.196.194:40213)
0925 02:59:43.944342 (+ 14491us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:43.944352 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:43.944382 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:59:43.944482 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:43.944491 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:43.945927 (+ 1436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:43.946112 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:59:43.954614 (+ 8502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:43.956180 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:43.958750 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:43.960272 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:43.960422 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:43.960572 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:43.963048 (+ 2476us) server_negotiation.cc:299] Negotiation successful
0925 02:59:43.963221 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14325,"thread_start_us":14241,"threads_started":1}
I20260925 02:59:43.963514 32678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:43.943160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37700 (local address 127.18.196.195:33053)
0925 02:59:43.952699 (+ 9539us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:43.952711 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:43.954340 (+ 1629us) server_negotiation.cc:407] Connection header received
0925 02:59:43.954582 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:43.954589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:43.954806 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:43.955017 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:59:43.955849 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:43.957472 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:43.959917 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:43.961428 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:43.961632 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:43.961785 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:43.963018 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0925 02:59:43.963187 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9301,"thread_start_us":9549,"threads_started":1}
I20260925 02:59:43.964389 32673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:43.929245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:40213 (local address 127.0.0.1:46410)
0925 02:59:43.942295 (+ 13050us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:43.942339 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:43.942517 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:43.953009 (+ 10492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:43.953016 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:43.953053 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:59:43.953648 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:43.953662 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:43.956377 (+ 2715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:43.956382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:43.958563 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:43.958579 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:43.960785 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:43.960791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:43.960973 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:43.961003 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 02:59:43.963819 (+ 2816us) client_negotiation.cc:240] Negotiation successful
0925 02:59:43.963966 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12788,"mutex_wait_us":3,"thread_start_us":12649,"threads_started":1}
I20260925 02:59:43.967155 32586 tablet_service.cc:1467] Processing CreateTablet for tablet f8a32c0788d74c44aa8b8b5c8d7d5dfe (DEFAULT_TABLE table=test-table [id=f463c8d4ff5147cc8a7ba8b21f63969d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:59:43.967921 32451 tablet_service.cc:1467] Processing CreateTablet for tablet f8a32c0788d74c44aa8b8b5c8d7d5dfe (DEFAULT_TABLE table=test-table [id=f463c8d4ff5147cc8a7ba8b21f63969d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:59:43.968482 32586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8a32c0788d74c44aa8b8b5c8d7d5dfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:59:43.969211 32451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8a32c0788d74c44aa8b8b5c8d7d5dfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:59:43.964342 32305 tablet_service.cc:1467] Processing CreateTablet for tablet f8a32c0788d74c44aa8b8b5c8d7d5dfe (DEFAULT_TABLE table=test-table [id=f463c8d4ff5147cc8a7ba8b21f63969d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:59:43.972015 32305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8a32c0788d74c44aa8b8b5c8d7d5dfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:59:43.991227 32680 tablet_bootstrap.cc:492] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e: Bootstrap starting.
I20260925 02:59:43.995615 32679 tablet_bootstrap.cc:492] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 9b484aa0c156475f8a2bdd6549940178: Bootstrap starting.
I20260925 02:59:43.998121 32680 tablet_bootstrap.cc:654] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e: Neither blocks nor log segments found. Creating new log.
I20260925 02:59:43.999236 32679 tablet_bootstrap.cc:654] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 9b484aa0c156475f8a2bdd6549940178: Neither blocks nor log segments found. Creating new log.
I20260925 02:59:43.999998 32680 log.cc:824] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:59:44.000512 32679 log.cc:824] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 9b484aa0c156475f8a2bdd6549940178: Log is configured to *not* fsync() on all Append() calls
I20260925 02:59:44.000965 32681 tablet_bootstrap.cc:492] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4: Bootstrap starting.
I20260925 02:59:44.004843 32681 tablet_bootstrap.cc:654] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4: Neither blocks nor log segments found. Creating new log.
I20260925 02:59:44.005990 32681 log.cc:824] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4: Log is configured to *not* fsync() on all Append() calls
I20260925 02:59:44.013002 32681 tablet_bootstrap.cc:492] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4: No bootstrap required, opened a new log
I20260925 02:59:44.013298 32681 ts_tablet_manager.cc:1397] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260925 02:59:44.018152 32680 tablet_bootstrap.cc:492] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e: No bootstrap required, opened a new log
I20260925 02:59:44.018435 32680 ts_tablet_manager.cc:1397] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.003s
I20260925 02:59:44.022933 32679 tablet_bootstrap.cc:492] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 9b484aa0c156475f8a2bdd6549940178: No bootstrap required, opened a new log
I20260925 02:59:44.023343 32679 ts_tablet_manager.cc:1397] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 9b484aa0c156475f8a2bdd6549940178: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.012s
I20260925 02:59:44.027978 32681 raft_consensus.cc:348] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b484aa0c156475f8a2bdd6549940178" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40213 } } peers { permanent_uuid: "e7634175f3874cddaac517c7a44b12e4" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 33053 } } peers { permanent_uuid: "30e09a0b8c9b405d9c89640992042b0e" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 34409 } }
I20260925 02:59:44.028627 32681 raft_consensus.cc:374] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:59:44.028788 32681 raft_consensus.cc:729] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7634175f3874cddaac517c7a44b12e4, State: Initialized, Role: FOLLOWER
I20260925 02:59:44.029619 32681 consensus_queue.cc:260] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b484aa0c156475f8a2bdd6549940178" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40213 } } peers { permanent_uuid: "e7634175f3874cddaac517c7a44b12e4" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 33053 } } peers { permanent_uuid: "30e09a0b8c9b405d9c89640992042b0e" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 34409 } }
I20260925 02:59:44.031548 32680 raft_consensus.cc:348] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b484aa0c156475f8a2bdd6549940178" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40213 } } peers { permanent_uuid: "e7634175f3874cddaac517c7a44b12e4" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 33053 } } peers { permanent_uuid: "30e09a0b8c9b405d9c89640992042b0e" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 34409 } }
I20260925 02:59:44.032148 32680 raft_consensus.cc:374] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:59:44.032361 32680 raft_consensus.cc:729] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e09a0b8c9b405d9c89640992042b0e, State: Initialized, Role: FOLLOWER
I20260925 02:59:44.033238 32680 consensus_queue.cc:260] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b484aa0c156475f8a2bdd6549940178" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40213 } } peers { permanent_uuid: "e7634175f3874cddaac517c7a44b12e4" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 33053 } } peers { permanent_uuid: "30e09a0b8c9b405d9c89640992042b0e" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 34409 } }
I20260925 02:59:44.034590 32679 raft_consensus.cc:348] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 9b484aa0c156475f8a2bdd6549940178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b484aa0c156475f8a2bdd6549940178" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40213 } } peers { permanent_uuid: "e7634175f3874cddaac517c7a44b12e4" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 33053 } } peers { permanent_uuid: "30e09a0b8c9b405d9c89640992042b0e" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 34409 } }
I20260925 02:59:44.035100 32679 raft_consensus.cc:374] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 9b484aa0c156475f8a2bdd6549940178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:59:44.035240 32679 raft_consensus.cc:729] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 9b484aa0c156475f8a2bdd6549940178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b484aa0c156475f8a2bdd6549940178, State: Initialized, Role: FOLLOWER
I20260925 02:59:44.035935 32679 consensus_queue.cc:260] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 9b484aa0c156475f8a2bdd6549940178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b484aa0c156475f8a2bdd6549940178" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40213 } } peers { permanent_uuid: "e7634175f3874cddaac517c7a44b12e4" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 33053 } } peers { permanent_uuid: "30e09a0b8c9b405d9c89640992042b0e" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 34409 } }
I20260925 02:59:44.038348 32681 ts_tablet_manager.cc:1428] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4: Time spent starting tablet: real 0.025s user 0.015s sys 0.004s
I20260925 02:59:44.038802 32651 heartbeater.cc:503] Master 127.18.196.254:39091 was elected leader, sending a full tablet report...
I20260925 02:59:44.047395 32370 heartbeater.cc:503] Master 127.18.196.254:39091 was elected leader, sending a full tablet report...
I20260925 02:59:44.048111 32680 ts_tablet_manager.cc:1428] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e: Time spent starting tablet: real 0.030s user 0.017s sys 0.010s
I20260925 02:59:44.049140 32517 heartbeater.cc:503] Master 127.18.196.254:39091 was elected leader, sending a full tablet report...
I20260925 02:59:44.049643 32679 ts_tablet_manager.cc:1428] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 9b484aa0c156475f8a2bdd6549940178: Time spent starting tablet: real 0.026s user 0.015s sys 0.009s
W20260925 02:59:44.056592 32518 tablet.cc:2369] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 9b484aa0c156475f8a2bdd6549940178: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:59:44.066381 32371 tablet.cc:2369] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:59:44.071393 32686 raft_consensus.cc:482] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:59:44.071694 32686 raft_consensus.cc:504] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b484aa0c156475f8a2bdd6549940178" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40213 } } peers { permanent_uuid: "e7634175f3874cddaac517c7a44b12e4" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 33053 } } peers { permanent_uuid: "30e09a0b8c9b405d9c89640992042b0e" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 34409 } }
I20260925 02:59:44.073367 32686 leader_election.cc:288] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b484aa0c156475f8a2bdd6549940178 (127.18.196.194:40213), e7634175f3874cddaac517c7a44b12e4 (127.18.196.195:33053)
I20260925 02:59:44.075693 32685 raft_consensus.cc:482] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:59:44.075994 32685 raft_consensus.cc:504] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b484aa0c156475f8a2bdd6549940178" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40213 } } peers { permanent_uuid: "e7634175f3874cddaac517c7a44b12e4" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 33053 } } peers { permanent_uuid: "30e09a0b8c9b405d9c89640992042b0e" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 34409 } }
I20260925 02:59:44.078200 32685 leader_election.cc:288] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b484aa0c156475f8a2bdd6549940178 (127.18.196.194:40213), 30e09a0b8c9b405d9c89640992042b0e (127.18.196.193:34409)
I20260925 02:59:44.083817 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:44.073781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:46739 (local address 127.18.196.194:40213)
0925 02:59:44.074003 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:44.074011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:44.074377 (+ 366us) server_negotiation.cc:407] Connection header received
0925 02:59:44.074664 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:44.074674 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:44.074777 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:44.074876 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:59:44.075752 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:44.076847 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:44.079904 (+ 3057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:44.081365 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:44.081518 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:44.081609 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:44.083390 (+ 1781us) server_negotiation.cc:299] Negotiation successful
0925 02:59:44.083550 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":67}
I20260925 02:59:44.084703 32688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:44.073594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:40213 (local address 127.18.196.193:46739)
0925 02:59:44.074248 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:44.074295 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:44.074435 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:44.075095 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:44.075100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:44.075126 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:59:44.075563 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:44.075577 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:44.077031 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:44.077036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:44.079510 (+ 2474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:44.079529 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:44.083015 (+ 3486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:44.083021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:44.083207 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:44.083240 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 02:59:44.084207 (+ 967us) client_negotiation.cc:240] Negotiation successful
0925 02:59:44.084342 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":251,"thread_start_us":352,"threads_started":1}
I20260925 02:59:44.086364 32678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:44.074695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:43861 (local address 127.18.196.195:33053)
0925 02:59:44.074885 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:44.074890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:44.075836 (+ 946us) server_negotiation.cc:407] Connection header received
0925 02:59:44.076061 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:44.076065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:44.076143 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:44.076198 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:59:44.076893 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:44.077854 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:44.081371 (+ 3517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:44.082661 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:44.082800 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:44.082883 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:44.085966 (+ 3083us) server_negotiation.cc:299] Negotiation successful
0925 02:59:44.086117 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:59:44.086272 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a32c0788d74c44aa8b8b5c8d7d5dfe" candidate_uuid: "30e09a0b8c9b405d9c89640992042b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b484aa0c156475f8a2bdd6549940178" is_pre_election: true
I20260925 02:59:44.086851 32471 raft_consensus.cc:2454] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 9b484aa0c156475f8a2bdd6549940178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30e09a0b8c9b405d9c89640992042b0e in term 0.
I20260925 02:59:44.087580 32259 leader_election.cc:302] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e09a0b8c9b405d9c89640992042b0e, 9b484aa0c156475f8a2bdd6549940178; no voters:
I20260925 02:59:44.087975 32686 raft_consensus.cc:2781] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:59:44.088138 32686 raft_consensus.cc:482] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:59:44.088183 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:44.074410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:33053 (local address 127.18.196.193:43861)
0925 02:59:44.075738 (+ 1328us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:44.075759 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:44.075922 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:44.076361 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:44.076365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:44.076386 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:59:44.076726 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:44.076737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:44.078021 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:44.078025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:44.081164 (+ 3139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:44.081181 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:44.085070 (+ 3889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:44.085076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:44.085251 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:44.085280 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 02:59:44.086744 (+ 1464us) client_negotiation.cc:240] Negotiation successful
0925 02:59:44.086912 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1226,"mutex_wait_us":3,"thread_start_us":845,"threads_started":1}
I20260925 02:59:44.088251 32686 raft_consensus.cc:3037] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:59:44.089028 32606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a32c0788d74c44aa8b8b5c8d7d5dfe" candidate_uuid: "30e09a0b8c9b405d9c89640992042b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7634175f3874cddaac517c7a44b12e4" is_pre_election: true
I20260925 02:59:44.089505 32606 raft_consensus.cc:2454] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30e09a0b8c9b405d9c89640992042b0e in term 0.
I20260925 02:59:44.089777 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:44.078982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:57887 (local address 127.18.196.194:40213)
0925 02:59:44.079712 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:44.079717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:44.079738 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:59:44.079796 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:44.079800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:44.079879 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:44.080416 (+ 537us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:59:44.080685 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:44.081907 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:44.084709 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:44.088270 (+ 3561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:44.089208 (+ 938us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:44.089276 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:44.089341 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 02:59:44.089474 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":4009,"threads_started":1}
I20260925 02:59:44.090612 32655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:44.078643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:40213 (local address 127.18.196.195:57887)
0925 02:59:44.078933 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:44.078949 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:44.079038 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:44.080029 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:44.080034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:44.080061 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:59:44.080282 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:44.080293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:44.082066 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:44.082070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:44.084412 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:44.084432 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:44.088457 (+ 4025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:44.088465 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:44.088670 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:44.088708 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 02:59:44.088785 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 02:59:44.088913 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260925 02:59:44.091848 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a32c0788d74c44aa8b8b5c8d7d5dfe" candidate_uuid: "e7634175f3874cddaac517c7a44b12e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b484aa0c156475f8a2bdd6549940178" is_pre_election: true
I20260925 02:59:44.092319 32471 raft_consensus.cc:2454] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 9b484aa0c156475f8a2bdd6549940178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7634175f3874cddaac517c7a44b12e4 in term 0.
I20260925 02:59:44.092139 32686 raft_consensus.cc:504] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b484aa0c156475f8a2bdd6549940178" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40213 } } peers { permanent_uuid: "e7634175f3874cddaac517c7a44b12e4" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 33053 } } peers { permanent_uuid: "30e09a0b8c9b405d9c89640992042b0e" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 34409 } }
I20260925 02:59:44.092955 32540 leader_election.cc:302] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b484aa0c156475f8a2bdd6549940178, e7634175f3874cddaac517c7a44b12e4; no voters:
I20260925 02:59:44.093384 32685 raft_consensus.cc:2781] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:59:44.093511 32685 raft_consensus.cc:482] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:59:44.093602 32685 raft_consensus.cc:3037] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:59:44.093969 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a32c0788d74c44aa8b8b5c8d7d5dfe" candidate_uuid: "30e09a0b8c9b405d9c89640992042b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b484aa0c156475f8a2bdd6549940178"
I20260925 02:59:44.094230 32471 raft_consensus.cc:3037] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 9b484aa0c156475f8a2bdd6549940178 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:59:44.096726 32685 raft_consensus.cc:504] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b484aa0c156475f8a2bdd6549940178" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40213 } } peers { permanent_uuid: "e7634175f3874cddaac517c7a44b12e4" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 33053 } } peers { permanent_uuid: "30e09a0b8c9b405d9c89640992042b0e" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 34409 } }
I20260925 02:59:44.097415 32685 leader_election.cc:288] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4 [CANDIDATE]: Term 1 election: Requested vote from peers 9b484aa0c156475f8a2bdd6549940178 (127.18.196.194:40213), 30e09a0b8c9b405d9c89640992042b0e (127.18.196.193:34409)
I20260925 02:59:44.098184 32686 leader_election.cc:288] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e [CANDIDATE]: Term 1 election: Requested vote from peers 9b484aa0c156475f8a2bdd6549940178 (127.18.196.194:40213), e7634175f3874cddaac517c7a44b12e4 (127.18.196.195:33053)
I20260925 02:59:44.099050 32471 raft_consensus.cc:2454] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 9b484aa0c156475f8a2bdd6549940178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30e09a0b8c9b405d9c89640992042b0e in term 1.
I20260925 02:59:44.098964 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a32c0788d74c44aa8b8b5c8d7d5dfe" candidate_uuid: "e7634175f3874cddaac517c7a44b12e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b484aa0c156475f8a2bdd6549940178"
I20260925 02:59:44.099094 32606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a32c0788d74c44aa8b8b5c8d7d5dfe" candidate_uuid: "30e09a0b8c9b405d9c89640992042b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7634175f3874cddaac517c7a44b12e4"
I20260925 02:59:44.099386 32606 raft_consensus.cc:2379] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 30e09a0b8c9b405d9c89640992042b0e in current term 1: Already voted for candidate e7634175f3874cddaac517c7a44b12e4 in this term.
I20260925 02:59:44.099416 32470 raft_consensus.cc:2379] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 9b484aa0c156475f8a2bdd6549940178 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e7634175f3874cddaac517c7a44b12e4 in current term 1: Already voted for candidate 30e09a0b8c9b405d9c89640992042b0e in this term.
I20260925 02:59:44.099627 32259 leader_election.cc:302] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e09a0b8c9b405d9c89640992042b0e, 9b484aa0c156475f8a2bdd6549940178; no voters:
I20260925 02:59:44.100507 32686 raft_consensus.cc:2781] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:59:44.100971 32686 raft_consensus.cc:686] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e [term 1 LEADER]: Becoming Leader. State: Replica: 30e09a0b8c9b405d9c89640992042b0e, State: Running, Role: LEADER
I20260925 02:59:44.101467 32655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:44.079712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:34409 (local address 127.18.196.195:57417)
0925 02:59:44.090813 (+ 11101us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:44.090856 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:44.090956 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:44.094945 (+ 3989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:44.094954 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:44.095000 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:59:44.095270 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:44.095284 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:44.096799 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:44.096805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:44.098706 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:44.098723 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:44.100524 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:44.100528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:44.100816 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:44.100864 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 02:59:44.100968 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 02:59:44.101115 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11000,"thread_start_us":16374,"threads_started":1}
I20260925 02:59:44.101547 32686 consensus_queue.cc:237] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b484aa0c156475f8a2bdd6549940178" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40213 } } peers { permanent_uuid: "e7634175f3874cddaac517c7a44b12e4" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 33053 } } peers { permanent_uuid: "30e09a0b8c9b405d9c89640992042b0e" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 34409 } }
I20260925 02:59:44.102152 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:44.079333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:57417 (local address 127.18.196.193:34409)
0925 02:59:44.079537 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:44.079555 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:44.093082 (+ 13527us) server_negotiation.cc:407] Connection header received
0925 02:59:44.093163 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:44.093172 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:44.093272 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:44.093392 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:59:44.095497 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:44.096629 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:44.098918 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:44.100201 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:44.100346 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:44.100417 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:44.101761 (+ 1344us) server_negotiation.cc:299] Negotiation successful
0925 02:59:44.101930 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 02:59:44.103176 32325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a32c0788d74c44aa8b8b5c8d7d5dfe" candidate_uuid: "e7634175f3874cddaac517c7a44b12e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e09a0b8c9b405d9c89640992042b0e" is_pre_election: true
I20260925 02:59:44.103334 32324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a32c0788d74c44aa8b8b5c8d7d5dfe" candidate_uuid: "e7634175f3874cddaac517c7a44b12e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e09a0b8c9b405d9c89640992042b0e"
W20260925 02:59:44.103832 32652 tablet.cc:2369] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:59:44.104310 32541 leader_election.cc:302] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7634175f3874cddaac517c7a44b12e4; no voters: 30e09a0b8c9b405d9c89640992042b0e, 9b484aa0c156475f8a2bdd6549940178
I20260925 02:59:44.104597 32685 raft_consensus.cc:2726] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 02:59:44.108188 32171 catalog_manager.cc:5495] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 30e09a0b8c9b405d9c89640992042b0e (127.18.196.193). New cstate: current_term: 1 leader_uuid: "30e09a0b8c9b405d9c89640992042b0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b484aa0c156475f8a2bdd6549940178" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7634175f3874cddaac517c7a44b12e4" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 33053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30e09a0b8c9b405d9c89640992042b0e" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 34409 } health_report { overall_health: HEALTHY } } }
I20260925 02:59:44.133854 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:44.127446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52318 (local address 127.18.196.193:34409)
0925 02:59:44.127628 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:44.127634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:44.127700 (+ 66us) server_negotiation.cc:407] Connection header received
0925 02:59:44.127946 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:44.127952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:44.128053 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:44.128250 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:59:44.128577 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:44.129755 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:44.131160 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:44.131789 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:44.131914 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:44.132387 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:59:44.133083 (+ 696us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:59:44.133249 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:44.133461 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 02:59:44.133548 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:59:44.142246 32471 raft_consensus.cc:1261] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 9b484aa0c156475f8a2bdd6549940178 [term 1 FOLLOWER]: Refusing update from remote peer 30e09a0b8c9b405d9c89640992042b0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:59:44.142385 32606 raft_consensus.cc:1261] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4 [term 1 FOLLOWER]: Refusing update from remote peer 30e09a0b8c9b405d9c89640992042b0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:59:44.143203 32686 consensus_queue.cc:1035] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b484aa0c156475f8a2bdd6549940178" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 40213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:59:44.143627 32692 consensus_queue.cc:1035] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7634175f3874cddaac517c7a44b12e4" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 33053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:59:44.160838 32695 mvcc.cc:205] Tried to move back new op lower bound from 7333090034236772352 to 7333090034084454400. Current Snapshot: MvccSnapshot[applied={T|T < 7333090034236772352}]
I20260925 02:59:44.188798 32173 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:59:44.189116 32173 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:59:44.193418 32173 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08903717dd064378b76fbb1fb6e9969d: Sending DeleteTablet for 3 replicas of tablet f8a32c0788d74c44aa8b8b5c8d7d5dfe
I20260925 02:59:44.194811 32586 tablet_service.cc:1514] Processing DeleteTablet for tablet f8a32c0788d74c44aa8b8b5c8d7d5dfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:59:44 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:37700
I20260925 02:59:44.195323 32451 tablet_service.cc:1514] Processing DeleteTablet for tablet f8a32c0788d74c44aa8b8b5c8d7d5dfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:59:44 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:46410
I20260925 02:59:44.195353 32305 tablet_service.cc:1514] Processing DeleteTablet for tablet f8a32c0788d74c44aa8b8b5c8d7d5dfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:59:44 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:52304
I20260925 02:59:44.203665 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 32210
I20260925 02:59:44.231027 32707 tablet_replica.cc:330] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 9b484aa0c156475f8a2bdd6549940178: stopping tablet replica
I20260925 02:59:44.231633 32707 raft_consensus.cc:2229] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 9b484aa0c156475f8a2bdd6549940178 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:59:44.232308 32707 raft_consensus.cc:2258] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 9b484aa0c156475f8a2bdd6549940178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:59:44.233696 32705 tablet_replica.cc:330] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4: stopping tablet replica
I20260925 02:59:44.235656 32707 ts_tablet_manager.cc:1905] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 9b484aa0c156475f8a2bdd6549940178: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:59:44.235878 32706 tablet_replica.cc:330] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e: stopping tablet replica
I20260925 02:59:44.236313 32706 raft_consensus.cc:2229] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:59:44.237128 32706 raft_consensus.cc:2258] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:59:44.236405 32705 raft_consensus.cc:2229] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:59:44.237946 32705 raft_consensus.cc:2258] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:59:44.238143 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:44.205238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52322 (local address 127.18.196.193:34409)
0925 02:59:44.205885 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:44.205891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:44.205909 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:59:44.205970 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:44.205976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:44.206058 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:44.206158 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:44.208195 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:44.209371 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:44.213203 (+ 3832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:44.213903 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:44.214036 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:44.220918 (+ 6882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:44.220936 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:44.220967 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:44.221025 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:44.226369 (+ 5344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:44.227971 (+ 1602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:44.227981 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:44.227985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:44.228524 (+ 539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:44.229669 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:44.229675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:44.229679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:44.235921 (+ 6242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:44.236277 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:44.237569 (+ 1292us) server_negotiation.cc:299] Negotiation successful
0925 02:59:44.237816 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501}
I20260925 02:59:44.238925 32365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:59:44.244379 32705 ts_tablet_manager.cc:1905] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:59:44.248586 32707 ts_tablet_manager.cc:1918] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 9b484aa0c156475f8a2bdd6549940178: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:59:44.248863 32707 log.cc:1196] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 9b484aa0c156475f8a2bdd6549940178: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/f8a32c0788d74c44aa8b8b5c8d7d5dfe
I20260925 02:59:44.249557 32707 ts_tablet_manager.cc:1939] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 9b484aa0c156475f8a2bdd6549940178: Deleting consensus metadata
I20260925 02:59:44.251875 32158 catalog_manager.cc:4841] TS 9b484aa0c156475f8a2bdd6549940178 (127.18.196.194:40213): tablet f8a32c0788d74c44aa8b8b5c8d7d5dfe (table test-table [id=f463c8d4ff5147cc8a7ba8b21f63969d]) successfully deleted
I20260925 02:59:44.252962 32705 ts_tablet_manager.cc:1918] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:59:44.253122 32705 log.cc:1196] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/f8a32c0788d74c44aa8b8b5c8d7d5dfe
I20260925 02:59:44.253687 32705 ts_tablet_manager.cc:1939] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P e7634175f3874cddaac517c7a44b12e4: Deleting consensus metadata
I20260925 02:59:44.255456 32158 catalog_manager.cc:4841] TS e7634175f3874cddaac517c7a44b12e4 (127.18.196.195:33053): tablet f8a32c0788d74c44aa8b8b5c8d7d5dfe (table test-table [id=f463c8d4ff5147cc8a7ba8b21f63969d]) successfully deleted
I20260925 02:59:44.347393 32706 ts_tablet_manager.cc:1905] T f8a32c0788d74c44aa8b8b5c8d7d5dfe P 30e09a0b8c9b405d9c89640992042b0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:59:44.348469 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 32210
W20260925 02:59:44.368114 32159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.196.193:34409 (error 108)
I20260925 02:59:44.368227 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 32374
I20260925 02:59:44.370106 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:44.369652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:34409 (local address 127.0.0.1:52334)
0925 02:59:44.369834 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:44.369963 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:34409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 02:59:44.370507 32159 catalog_manager.cc:4584] TS 30e09a0b8c9b405d9c89640992042b0e (127.18.196.193:34409): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f8a32c0788d74c44aa8b8b5c8d7d5dfe: Network error: Client connection negotiation failed: client connection to 127.18.196.193:34409: connect: Connection refused (error 111)
I20260925 02:59:44.386442 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:44.369049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46414 (local address 127.18.196.194:40213)
0925 02:59:44.369256 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:44.369263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:44.369372 (+ 109us) server_negotiation.cc:407] Connection header received
0925 02:59:44.369555 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:44.369559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:44.369632 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:44.369754 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:44.371624 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:44.372324 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:44.373508 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:44.373917 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:44.374057 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:44.381251 (+ 7194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:44.381267 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:44.381284 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:44.381321 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:44.384139 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:44.384734 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:44.384741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:44.384743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:44.384889 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:44.385207 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:44.385210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:44.385212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:44.385543 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:44.385760 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:44.386014 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 02:59:44.386198 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 02:59:44.387183 32512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:59:44.424364 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:44.423763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:34409 (local address 127.0.0.1:52338)
0925 02:59:44.424007 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:44.424140 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:34409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 02:59:44.425441 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:44.425051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:34409 (local address 127.0.0.1:52340)
0925 02:59:44.425205 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:44.425299 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:34409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 02:59:44.485255 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 32374
I20260925 02:59:44.486328 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:44.485406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:34409 (local address 127.0.0.1:52342)
0925 02:59:44.486000 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:44.486115 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:34409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420}
I20260925 02:59:44.487587 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:44.487226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:34409 (local address 127.0.0.1:52344)
0925 02:59:44.487364 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:44.487437 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:34409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 02:59:44.503605 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 32521
I20260925 02:59:44.517798 32678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:44.504414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37714 (local address 127.18.196.195:33053)
0925 02:59:44.504601 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:44.504608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:44.504695 (+ 87us) server_negotiation.cc:407] Connection header received
0925 02:59:44.504925 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:44.504930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:44.504996 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:44.505105 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:44.506393 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:44.507230 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:44.508493 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:44.508912 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:44.509048 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:44.511736 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:44.511751 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:44.511767 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:44.511806 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:44.514634 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:44.515409 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:44.515416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:44.515418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:44.515565 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:44.516152 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:44.516160 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:44.516163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:44.516536 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:44.516832 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:44.517224 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 02:59:44.517406 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:59:44.518505 32646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:59:44.568439 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:44.567832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:34409 (local address 127.0.0.1:52350)
0925 02:59:44.568111 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:44.568232 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:34409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260925 02:59:44.569571 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:44.569132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:34409 (local address 127.0.0.1:52364)
0925 02:59:44.569350 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:44.569437 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:34409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260925 02:59:44.610872 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 32521
I20260925 02:59:44.628418 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 32140
I20260925 02:59:44.629709 32203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:59:44.742060 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:44.741516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:34409 (local address 127.0.0.1:52376)
0925 02:59:44.741732 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:44.741854 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:34409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 02:59:44.742094 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 32140
I20260925 02:59:44.766884 19219 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 02:59:45.102138 19219 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 02:59:45.102391 19219 mini_ranger.cc:66] Stopping Ranger...
I20260925 02:59:45.447388 19219 mini_ranger.cc:68] Stopped Ranger
2026-09-25 02:59:45.447 UTC [31191] LOG: received smart shutdown request
2026-09-25 02:59:45.450 UTC [31191] LOG: background worker "logical replication launcher" (PID 31200) exited with exit code 1
2026-09-25 02:59:45.451 UTC [31195] LOG: shutting down
2026-09-25 02:59:45.618 UTC [31191] LOG: database system is shut down
2026-09-25T02:59:45Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50849 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:59:45 dist-test-slave-8zvk krb5kdc[32721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:59:45 dist-test-slave-8zvk krb5kdc[32721](info): set up 2 sockets
Sep 25 02:59:45 dist-test-slave-8zvk krb5kdc[32721](info): commencing operation
krb5kdc: starting...
W20260925 02:59:46.834004 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 02:59:46 dist-test-slave-8zvk krb5kdc[32721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:59:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:59:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 02:59:47.065837 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:36261
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:39887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:36261,127.18.196.253:43901,127.18.196.252:35691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:59:47.387003 32734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:59:47.387315 32734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:59:47.387477 32734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:59:47.394230 32734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:59:47.394359 32734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:59:47.394444 32734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:59:47.394529 32734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:59:47.394593 32734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:59:47.403734 32734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:39887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:36261,127.18.196.253:43901,127.18.196.252:35691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:36261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:59:47.404922 32734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:59:47.406105 32734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:59:47.416965 32739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:47.416998 32740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:47.417042 32742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:59:47.417965 32734 server_base.cc:1034] running on GCE node
I20260925 02:59:47.418737 32734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:59:47.420063 32734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:59:47.421255 32734 hybrid_clock.cc:648] HybridClock initialized: now 1790305187421223 us; error 25 us; skew 500 ppm
I20260925 02:59:47.421562 32734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:59:47 dist-test-slave-8zvk krb5kdc[32721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305187, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:59:47.424479 32734 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 02:59:47.425864 32734 webserver.cc:460] Webserver started at http://127.18.196.254:38211/ using document root <none> and password file <none>
I20260925 02:59:47.426509 32734 fs_manager.cc:359] Metadata directory not provided
I20260925 02:59:47.426645 32734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:59:47.426988 32734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:59:47.429726 32734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "08ea2c62c528446181724e48a6256f9a"
format_stamp: "Formatted at 2026-09-25 02:59:47 on dist-test-slave-8zvk"
server_key: "4805e4a6987b1f68732826502e7ad219"
server_key_iv: "10debef2820b6d34bc6d100fce35cc4b"
server_key_version: "encryptionkey@0"
I20260925 02:59:47.430505 32734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "08ea2c62c528446181724e48a6256f9a"
format_stamp: "Formatted at 2026-09-25 02:59:47 on dist-test-slave-8zvk"
server_key: "4805e4a6987b1f68732826502e7ad219"
server_key_iv: "10debef2820b6d34bc6d100fce35cc4b"
server_key_version: "encryptionkey@0"
I20260925 02:59:47.435160 32734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260925 02:59:47.438686 32749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:47.439935 32734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:59:47.440102 32734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "08ea2c62c528446181724e48a6256f9a"
format_stamp: "Formatted at 2026-09-25 02:59:47 on dist-test-slave-8zvk"
server_key: "4805e4a6987b1f68732826502e7ad219"
server_key_iv: "10debef2820b6d34bc6d100fce35cc4b"
server_key_version: "encryptionkey@0"
I20260925 02:59:47.440287 32734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:59:47.462568 32734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:59:47.466602 32734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:59:47.466936 32734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:59:47.485484 32734 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:36261
I20260925 02:59:47.485527 332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:36261 every 8 connection(s)
I20260925 02:59:47.486766 32734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:59:47.490342 333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:59:47.496398 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 32734
I20260925 02:59:47.496622 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:59:47.497255 19219 external_mini_cluster.cc:1428] Setting key 622fce8cb251354259020c7a0450f833
I20260925 02:59:47.496753 333 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36261 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:59:47 dist-test-slave-8zvk krb5kdc[32721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305187, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.196.253@KRBTEST.COM; defaulting to no policy
I20260925 02:59:47.517972 336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:47.501125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44286 (local address 127.18.196.254:36261)
0925 02:59:47.501855 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:47.501863 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:47.501929 (+ 66us) server_negotiation.cc:407] Connection header received
0925 02:59:47.503077 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:47.503082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:47.503335 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:47.503560 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:47.504794 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:47.505942 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:47.507365 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:47.507897 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:47.508118 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:47.511302 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:47.511309 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:47.511319 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:47.511344 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:47.513815 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:47.514473 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:47.514478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:47.514480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:47.514616 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:47.515080 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:47.515090 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:47.515092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:47.515538 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:47.515752 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:47.516276 (+ 524us) server_negotiation.cc:299] Negotiation successful
0925 02:59:47.516570 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":315,"threads_started":1}
I20260925 02:59:47.517940 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:47.499870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:36261 (local address 127.0.0.1:44286)
0925 02:59:47.500622 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:47.500658 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:47.502669 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:47.503587 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:47.503594 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:47.504274 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:59:47.504594 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:47.504610 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:47.506094 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:47.506100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:47.507226 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:47.507239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:47.508135 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:47.508142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:47.508337 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:47.508982 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:59:47.509002 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:59:47.511080 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:59:47.513944 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:47.513956 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:47.513966 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:47.514335 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:47.514730 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:47.514735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:47.514739 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:47.514966 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:47.515673 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:59:47.515692 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:59:47.515901 (+ 209us) client_negotiation.cc:769] Sending connection context
0925 02:59:47.516131 (+ 230us) client_negotiation.cc:240] Negotiation successful
0925 02:59:47.516673 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":124,"thread_start_us":329,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.253@KRBTEST.COM" created.
I20260925 02:59:47.520897 333 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:59:47.521891 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:47.521446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:43901 (local address 127.0.0.1:48514)
0925 02:59:47.521627 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:47.521790 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:43901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 02:59:47.522131 32752 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.196.253:43901: connect: Connection refused (error 111)
I20260925 02:59:47.523622 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:47.523302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:43901 (local address 127.0.0.1:48516)
0925 02:59:47.523445 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:47.523513 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:43901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 02:59:47.523792 333 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.253:43901: Network error: Client connection negotiation failed: client connection to 127.18.196.253:43901: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.253.keytab.
Entry for principal kudu/127.18.196.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.253.keytab.
I20260925 02:59:47.546353 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.253:43901
--webserver_interface=127.18.196.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.253.keytab
--principal=kudu/127.18.196.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:39887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:36261,127.18.196.253:43901,127.18.196.252:35691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:59:47.573207 333 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } attempt: 1
I20260925 02:59:47.574414 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:47.573973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:43901 (local address 127.0.0.1:48526)
0925 02:59:47.574168 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:47.574285 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:43901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 02:59:47.575444 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:47.575085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:43901 (local address 127.0.0.1:48534)
0925 02:59:47.575240 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:47.575318 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:43901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 02:59:47.575637 333 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.253:43901: Network error: Client connection negotiation failed: client connection to 127.18.196.253:43901: connect: Connection refused (error 111)
I20260925 02:59:47.643908 333 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } attempt: 2
I20260925 02:59:47.645129 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:47.644594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:43901 (local address 127.0.0.1:48540)
0925 02:59:47.644860 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:47.644947 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:43901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 02:59:47.646258 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:47.645877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:43901 (local address 127.0.0.1:48548)
0925 02:59:47.646058 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:47.646139 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:43901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260925 02:59:47.646447 333 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.253:43901: Network error: Client connection negotiation failed: client connection to 127.18.196.253:43901: connect: Connection refused (error 111)
I20260925 02:59:47.737697 333 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } attempt: 3
I20260925 02:59:47.738878 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:47.738389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:43901 (local address 127.0.0.1:48564)
0925 02:59:47.738594 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:47.738704 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:43901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 02:59:47.739908 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:47.739557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:43901 (local address 127.0.0.1:48566)
0925 02:59:47.739706 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:47.739778 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:43901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 02:59:47.740099 333 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.253:43901: Network error: Client connection negotiation failed: client connection to 127.18.196.253:43901: connect: Connection refused (error 111)
W20260925 02:59:47.866762 340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:59:47.867122 340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:59:47.867285 340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:59:47.874233 340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:59:47.874383 340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:59:47.874488 340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:59:47.874598 340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:59:47.874683 340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:59:47.883332 333 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } attempt: 4
I20260925 02:59:47.884574 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:47.884002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:43901 (local address 127.0.0.1:48572)
0925 02:59:47.884353 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:47.884442 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:43901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20260925 02:59:47.884979 340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:39887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:36261,127.18.196.253:43901,127.18.196.252:35691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.253.keytab
--principal=kudu/127.18.196.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.253:43901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.196.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:59:47.885623 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:47.885273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:43901 (local address 127.0.0.1:48578)
0925 02:59:47.885417 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:47.885511 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:43901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 02:59:47.885813 333 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.253:43901: Network error: Client connection negotiation failed: client connection to 127.18.196.253:43901: connect: Connection refused (error 111)
I20260925 02:59:47.886400 340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:59:47.887771 340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:59:47.898572 348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:47.898614 345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:47.898655 346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:59:47.904069 340 server_base.cc:1034] running on GCE node
I20260925 02:59:47.904819 340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:59:47.906061 340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:59:47.907253 340 hybrid_clock.cc:648] HybridClock initialized: now 1790305187907216 us; error 25 us; skew 500 ppm
I20260925 02:59:47.907575 340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:59:47 dist-test-slave-8zvk krb5kdc[32721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305187, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:59:47.916036 340 init.cc:377] Logged in from keytab as kudu/127.18.196.253@KRBTEST.COM (short username kudu)
I20260925 02:59:47.917392 340 webserver.cc:460] Webserver started at http://127.18.196.253:38187/ using document root <none> and password file <none>
I20260925 02:59:47.918011 340 fs_manager.cc:359] Metadata directory not provided
I20260925 02:59:47.918110 340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:59:47.918360 340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:59:47.921227 340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-1/data/instance:
uuid: "022251c42cdd49d28708b5c08850d24c"
format_stamp: "Formatted at 2026-09-25 02:59:47 on dist-test-slave-8zvk"
server_key: "8c34ab82bbb8394f7440678709f1cb05"
server_key_iv: "71f6cf452f683e3c2dc03c437db6e256"
server_key_version: "encryptionkey@0"
I20260925 02:59:47.921968 340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-1/wal/instance:
uuid: "022251c42cdd49d28708b5c08850d24c"
format_stamp: "Formatted at 2026-09-25 02:59:47 on dist-test-slave-8zvk"
server_key: "8c34ab82bbb8394f7440678709f1cb05"
server_key_iv: "71f6cf452f683e3c2dc03c437db6e256"
server_key_version: "encryptionkey@0"
I20260925 02:59:47.926721 340 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 02:59:47.929862 355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:47.931141 340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:59:47.931316 340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-1/wal
uuid: "022251c42cdd49d28708b5c08850d24c"
format_stamp: "Formatted at 2026-09-25 02:59:47 on dist-test-slave-8zvk"
server_key: "8c34ab82bbb8394f7440678709f1cb05"
server_key_iv: "71f6cf452f683e3c2dc03c437db6e256"
server_key_version: "encryptionkey@0"
I20260925 02:59:47.931523 340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:59:47.958261 340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:59:47.962047 340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:59:47.962337 340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:59:47.980086 340 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.253:43901
I20260925 02:59:47.980124 406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.253:43901 every 8 connection(s)
I20260925 02:59:47.981537 340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-1/data/info.pb
I20260925 02:59:47.985203 407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:59:47.988392 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 340
I20260925 02:59:47.988580 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-1/wal/instance
I20260925 02:59:47.989116 19219 external_mini_cluster.cc:1428] Setting key a61e81a8919213655e6a4dad23dbe12f
I20260925 02:59:47.993664 407 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36261 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:59:48 dist-test-slave-8zvk krb5kdc[32721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305187, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.253@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.196.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.252@KRBTEST.COM" created.
I20260925 02:59:48.017843 336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:47.996829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44290 (local address 127.18.196.254:36261)
0925 02:59:47.997054 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:47.997061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:47.998344 (+ 1283us) server_negotiation.cc:407] Connection header received
0925 02:59:48.001021 (+ 2677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:48.001025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:48.001091 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:48.001183 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:48.003279 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.003941 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.006398 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.006899 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.007015 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:48.011430 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:48.011435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:48.011438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:48.011473 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:48.014293 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.015258 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.015262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.015265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.015384 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.015828 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.015832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.015834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.016108 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:48.016271 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:48.017153 (+ 882us) server_negotiation.cc:299] Negotiation successful
0925 02:59:48.017401 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 02:59:48.018569 409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:47.997347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:36261 (local address 127.0.0.1:44290)
0925 02:59:47.998171 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:47.998222 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:48.000742 (+ 2520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:48.001457 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:48.001470 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:48.002256 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:59:48.003080 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:48.003108 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.004102 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.004110 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:48.006174 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:48.006192 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.007131 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.007140 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:48.007452 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:48.008172 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:59:48.008203 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:59:48.011145 (+ 2942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:59:48.014441 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:48.014451 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:48.014469 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:48.015109 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:48.015503 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:48.015508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:48.015512 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:48.015697 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:48.016244 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:59:48.016252 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:59:48.016681 (+ 429us) client_negotiation.cc:769] Sending connection context
0925 02:59:48.017003 (+ 322us) client_negotiation.cc:240] Negotiation successful
0925 02:59:48.017433 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":376,"threads_started":1}
I20260925 02:59:48.020277 407 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:59:48 dist-test-slave-8zvk krb5kdc[32721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305187, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.253@KRBTEST.COM for kudu/127.18.196.253@KRBTEST.COM
I20260925 02:59:48.034238 409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.020974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:43901 (local address 127.0.0.1:48590)
0925 02:59:48.021157 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.021171 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:48.021253 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:48.022331 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:48.022337 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:48.022763 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:59:48.022970 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:48.022980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.024266 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.024274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:48.025093 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:48.025104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.025905 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.025910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:48.026169 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:48.026724 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:59:48.026741 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:59:48.028660 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:59:48.031798 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:48.031806 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:48.031810 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:48.032155 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:48.032514 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:48.032520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:48.032523 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:48.032655 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:48.033260 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:59:48.033274 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:59:48.033419 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 02:59:48.033573 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 02:59:48.033835 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":3}
I20260925 02:59:48.034358 411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.021288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48590 (local address 127.18.196.253:43901)
0925 02:59:48.021949 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:48.021954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:48.021978 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:59:48.022018 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:48.022024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:48.022188 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:48.022344 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:48.023119 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.024131 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.025241 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.025762 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.025897 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:48.028881 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:48.028891 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:48.028903 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:48.028929 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:48.031659 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.032281 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.032289 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.032293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.032410 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.032753 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.032760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.032762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.033137 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:48.033358 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:48.033698 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 02:59:48.033938 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":523,"thread_start_us":388,"threads_started":1}
I20260925 02:59:48.036945 407 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:59:48.038108 409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.037622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:35691 (local address 127.0.0.1:57062)
0925 02:59:48.037862 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.037959 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260925 02:59:48.038458 359 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
I20260925 02:59:48.039858 409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.039567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:35691 (local address 127.0.0.1:57070)
0925 02:59:48.039703 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.039770 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 02:59:48.040215 407 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:35691: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.252.keytab.
Entry for principal kudu/127.18.196.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.252.keytab.
I20260925 02:59:48.046361 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.252:35691
--webserver_interface=127.18.196.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.252.keytab
--principal=kudu/127.18.196.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:39887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:36261,127.18.196.253:43901,127.18.196.252:35691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:59:48.089526 407 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } attempt: 1
I20260925 02:59:48.090709 409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.090246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:35691 (local address 127.0.0.1:57072)
0925 02:59:48.090440 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.090549 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 02:59:48.091725 409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.091404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:35691 (local address 127.0.0.1:57084)
0925 02:59:48.091550 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.091626 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 02:59:48.091961 407 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:35691: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
I20260925 02:59:48.160204 407 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } attempt: 2
I20260925 02:59:48.161441 409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.160961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:35691 (local address 127.0.0.1:57098)
0925 02:59:48.161181 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.161276 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 02:59:48.162551 409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.162234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:35691 (local address 127.0.0.1:57102)
0925 02:59:48.162366 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.162437 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 02:59:48.162815 407 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:35691: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
I20260925 02:59:48.185060 333 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } attempt: 5
Sep 25 02:59:48 dist-test-slave-8zvk krb5kdc[32721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305187, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for kudu/127.18.196.253@KRBTEST.COM
I20260925 02:59:48.198048 411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.185916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48592 (local address 127.18.196.253:43901)
0925 02:59:48.186106 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:48.186112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:48.186132 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:59:48.186310 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:48.186322 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:48.186386 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:48.186488 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:48.187326 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.188010 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.189142 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.189597 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.189724 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:48.192855 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:48.192860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:48.192863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:48.192894 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:48.195493 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.196129 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.196136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.196145 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.196271 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.196650 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.196656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.196659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.196949 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:48.197104 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:48.197478 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 02:59:48.197696 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:59:48.198052 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.185794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:43901 (local address 127.0.0.1:48592)
0925 02:59:48.185989 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.186011 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:48.186138 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:48.186526 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:48.186531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:48.186989 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:59:48.187168 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:48.187177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.188143 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.188151 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:48.188995 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:48.189006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.189751 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.189756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:48.189971 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:48.190586 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:59:48.190605 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:59:48.192649 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:59:48.195640 (+ 2991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:48.195646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:48.195650 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:48.196000 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:48.196383 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:48.196388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:48.196391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:48.196539 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:48.197067 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:59:48.197072 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:59:48.197221 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 02:59:48.197367 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 02:59:48.197690 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 02:59:48.199378 333 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:59:48.200507 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.200147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:35691 (local address 127.0.0.1:57106)
0925 02:59:48.200311 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.200385 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 02:59:48.201429 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.201112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:35691 (local address 127.0.0.1:57114)
0925 02:59:48.201267 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.201334 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 02:59:48.201660 333 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:35691: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
I20260925 02:59:48.252900 333 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } attempt: 1
I20260925 02:59:48.254171 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.253650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:35691 (local address 127.0.0.1:57116)
0925 02:59:48.253852 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.253967 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 02:59:48.254078 407 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } attempt: 3
I20260925 02:59:48.255309 409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.254874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:35691 (local address 127.0.0.1:57130)
0925 02:59:48.255026 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.255122 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 02:59:48.255873 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.255213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:35691 (local address 127.0.0.1:57146)
0925 02:59:48.255650 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.255737 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260925 02:59:48.256109 333 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:35691: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
I20260925 02:59:48.256453 409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.255985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:35691 (local address 127.0.0.1:57150)
0925 02:59:48.256247 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.256345 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260925 02:59:48.256691 407 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:35691: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
I20260925 02:59:48.324333 333 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } attempt: 2
I20260925 02:59:48.325371 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.324894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:35691 (local address 127.0.0.1:57152)
0925 02:59:48.325120 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.325206 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 02:59:48.326332 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.326028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:35691 (local address 127.0.0.1:57166)
0925 02:59:48.326162 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.326229 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 02:59:48.326555 333 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:35691: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
W20260925 02:59:48.374779 414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:59:48.375177 414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:59:48.375344 414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:59:48.382362 414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:59:48.382498 414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:59:48.382606 414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:59:48.382718 414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:59:48.382809 414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:59:48.392426 414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:39887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:36261,127.18.196.253:43901,127.18.196.252:35691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.252.keytab
--principal=kudu/127.18.196.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.252:35691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.196.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:59:48.393663 414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:59:48.394979 414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:59:48.399914 407 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } attempt: 4
I20260925 02:59:48.401211 409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.400572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:35691 (local address 127.0.0.1:57172)
0925 02:59:48.400894 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.401014 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260925 02:59:48.402179 409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.401872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:35691 (local address 127.0.0.1:57176)
0925 02:59:48.402015 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.402085 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 02:59:48.402371 407 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:35691: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
W20260925 02:59:48.407050 419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:48.407012 422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:48.407224 420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:59:48.407867 414 server_base.cc:1034] running on GCE node
I20260925 02:59:48.408624 414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:59:48.409884 414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:59:48.411110 414 hybrid_clock.cc:648] HybridClock initialized: now 1790305188411077 us; error 38 us; skew 500 ppm
I20260925 02:59:48.411427 414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:59:48 dist-test-slave-8zvk krb5kdc[32721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305188, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:59:48.414422 414 init.cc:377] Logged in from keytab as kudu/127.18.196.252@KRBTEST.COM (short username kudu)
I20260925 02:59:48.415869 414 webserver.cc:460] Webserver started at http://127.18.196.252:32927/ using document root <none> and password file <none>
I20260925 02:59:48.416505 414 fs_manager.cc:359] Metadata directory not provided
I20260925 02:59:48.416648 414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:59:48.416946 414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:59:48.419808 414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-2/data/instance:
uuid: "49d3d2ae4980489786b3b14e0bcd8f11"
format_stamp: "Formatted at 2026-09-25 02:59:48 on dist-test-slave-8zvk"
server_key: "559a841e7722c76b1e76e604a6d43151"
server_key_iv: "af863bb5ad7485e8bee46b337581b029"
server_key_version: "encryptionkey@0"
I20260925 02:59:48.420570 414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-2/wal/instance:
uuid: "49d3d2ae4980489786b3b14e0bcd8f11"
format_stamp: "Formatted at 2026-09-25 02:59:48 on dist-test-slave-8zvk"
server_key: "559a841e7722c76b1e76e604a6d43151"
server_key_iv: "af863bb5ad7485e8bee46b337581b029"
server_key_version: "encryptionkey@0"
I20260925 02:59:48.425383 414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:59:48.428635 429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:48.429958 414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:59:48.430132 414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-2/wal
uuid: "49d3d2ae4980489786b3b14e0bcd8f11"
format_stamp: "Formatted at 2026-09-25 02:59:48 on dist-test-slave-8zvk"
server_key: "559a841e7722c76b1e76e604a6d43151"
server_key_iv: "af863bb5ad7485e8bee46b337581b029"
server_key_version: "encryptionkey@0"
I20260925 02:59:48.430320 414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:59:48.432732 333 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } attempt: 3
I20260925 02:59:48.433673 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.433264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:35691 (local address 127.0.0.1:57182)
0925 02:59:48.433442 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.433527 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 02:59:48.434728 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.434324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:35691 (local address 127.0.0.1:57194)
0925 02:59:48.434523 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.434623 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260925 02:59:48.434954 333 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:35691: Network error: Client connection negotiation failed: client connection to 127.18.196.252:35691: connect: Connection refused (error 111)
I20260925 02:59:48.446696 414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:59:48.450309 414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:59:48.450570 414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:59:48.468538 414 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.252:35691
I20260925 02:59:48.468572 480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.252:35691 every 8 connection(s)
I20260925 02:59:48.469848 414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-2/data/info.pb
I20260925 02:59:48.473290 481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:59:48.477676 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 414
I20260925 02:59:48.477869 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304987644417-19219-0/minicluster-data/master-2/wal/instance
I20260925 02:59:48.478434 19219 external_mini_cluster.cc:1428] Setting key 7fb0ae345d08ed41345ccc2e8cfe1b7b
I20260925 02:59:48.480322 481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36261 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:59:48 dist-test-slave-8zvk krb5kdc[32721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305188, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.252@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
Sep 25 02:59:48 dist-test-slave-8zvk krb5kdc[32721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.252@KRBTEST.COM
I20260925 02:59:48.499168 336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.482740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44292 (local address 127.18.196.254:36261)
0925 02:59:48.482951 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:48.482957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:48.483534 (+ 577us) server_negotiation.cc:407] Connection header received
0925 02:59:48.483771 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:48.483776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:48.483838 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:48.483945 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:48.485302 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.486402 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.488474 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.489109 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.489244 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:48.492492 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:48.492499 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:48.492504 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:48.492542 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:48.495861 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.496619 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.496631 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.496634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.496765 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.497215 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.497220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.497223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.497583 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:48.497699 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:48.498561 (+ 862us) server_negotiation.cc:299] Negotiation successful
0925 02:59:48.498781 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 02:59:48.500229 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.480232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57204 (local address 127.18.196.252:35691)
0925 02:59:48.481027 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:48.481081 (+ 54us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:48.481132 (+ 51us) server_negotiation.cc:407] Connection header received
0925 02:59:48.483032 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:48.483052 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:48.483406 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:48.483755 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:48.485256 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.486479 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.487880 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.488551 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.488713 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:48.493421 (+ 4708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:48.493430 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:48.493443 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:48.493475 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:48.496753 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.497522 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.497527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.497529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.497660 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.498003 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.498008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.498011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.498420 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:48.498689 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:48.499006 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 02:59:48.499216 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":600,"thread_start_us":399,"threads_started":1}
I20260925 02:59:48.500249 485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.482666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:36261 (local address 127.0.0.1:44292)
0925 02:59:48.483384 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.483429 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:48.483572 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:48.484011 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:48.484018 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:48.484547 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:59:48.485061 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:48.485088 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.486597 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.486602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:48.488323 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:48.488336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.489351 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.489359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:48.489588 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:48.490147 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:59:48.490167 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:59:48.492297 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:59:48.496008 (+ 3711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:48.496020 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:48.496030 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:48.496485 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:48.496906 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:48.496911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:48.496914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:48.497086 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:48.497700 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:59:48.497710 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:59:48.498131 (+ 421us) client_negotiation.cc:769] Sending connection context
0925 02:59:48.498430 (+ 299us) client_negotiation.cc:240] Negotiation successful
0925 02:59:48.498886 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":1,"spinlock_wait_cycles":43776,"thread_start_us":341,"threads_started":1}
I20260925 02:59:48.502374 481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:59:48 dist-test-slave-8zvk krb5kdc[32721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305188, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.252@KRBTEST.COM for kudu/127.18.196.253@KRBTEST.COM
Sep 25 02:59:48 dist-test-slave-8zvk krb5kdc[32721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:59:48.516150 411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.503142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48604 (local address 127.18.196.253:43901)
0925 02:59:48.503336 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:48.503344 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:48.503364 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:59:48.503529 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:48.503537 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:48.503619 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:48.503715 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:48.504442 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.505443 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.506899 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.507529 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.507662 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:48.510746 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:48.510753 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:48.510760 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:48.510804 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:48.513528 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.514377 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.514381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.514383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.514502 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.514961 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.514965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.514966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.515203 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:48.515321 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:48.515729 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 02:59:48.515875 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:59:48.516300 485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.503040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:43901 (local address 127.0.0.1:48604)
0925 02:59:48.503263 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.503289 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:48.503399 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:48.503764 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:48.503769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:48.504160 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:59:48.504287 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:48.504295 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.505566 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.505572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:48.506727 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:48.506738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.507655 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.507659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:48.507795 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:48.508294 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:59:48.508317 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:59:48.510544 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:59:48.513701 (+ 3157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:48.513708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:48.513714 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:48.514260 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:48.514629 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:48.514633 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:48.514635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:48.514850 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:48.515329 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:59:48.515334 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:59:48.515480 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 02:59:48.515649 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 02:59:48.515917 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260925 02:59:48.517270 481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:59:48.519188 336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.503317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44306 (local address 127.18.196.254:36261)
0925 02:59:48.503506 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:48.503511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:48.503591 (+ 80us) server_negotiation.cc:407] Connection header received
0925 02:59:48.503765 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:48.503769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:48.503848 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:48.503945 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:48.505323 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.506328 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.507954 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.508563 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.508691 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:48.513127 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:48.513132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:48.513135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:48.513162 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:48.516242 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.517122 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.517126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.517128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.517308 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.517699 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.517704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.517708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.518040 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:48.518312 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:48.518614 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 02:59:48.518880 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 02:59:48 dist-test-slave-8zvk krb5kdc[32721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305188, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.252@KRBTEST.COM for kudu/127.18.196.252@KRBTEST.COM
Sep 25 02:59:48 dist-test-slave-8zvk krb5kdc[32721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.253@KRBTEST.COM
I20260925 02:59:48.531041 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.518228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57214 (local address 127.18.196.252:35691)
0925 02:59:48.518404 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:48.518408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:48.518619 (+ 211us) server_negotiation.cc:407] Connection header received
0925 02:59:48.518755 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:48.518762 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:48.518859 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:48.518954 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:48.519754 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.520642 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.521973 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.522482 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.522602 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:48.525736 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:48.525743 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:48.525748 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:48.525778 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:48.528430 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.529108 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.529113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.529117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.529239 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.529620 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.529625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.529628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.529885 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:48.530041 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:48.530357 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 02:59:48.530714 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":159,"server-negotiator.queue_time_us":67}
I20260925 02:59:48.531056 485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.518293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:35691 (local address 127.0.0.1:57214)
0925 02:59:48.518535 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.518552 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:48.518646 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:48.518978 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:48.518984 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:48.519444 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:59:48.519586 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:48.519594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.520758 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.520765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:48.521843 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:48.521855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.522621 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.522625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:48.522757 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:48.523330 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:59:48.523345 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:59:48.525526 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:59:48.528587 (+ 3061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:48.528592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:48.528596 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:48.528983 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:48.529344 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:48.529351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:48.529355 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:48.529500 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:48.530009 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:59:48.530013 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:59:48.530145 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 02:59:48.530284 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 02:59:48.530592 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":3}
I20260925 02:59:48.534436 411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.520807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48620 (local address 127.18.196.253:43901)
0925 02:59:48.520973 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:48.520979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:48.520995 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:59:48.521094 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:48.521101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:48.521163 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:48.521256 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:48.522341 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.523405 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.525025 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.525484 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.525592 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:48.529577 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:48.529582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:48.529585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:48.529612 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:48.532167 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.532726 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.532732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.532735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.532891 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.533218 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.533224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.533228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.533460 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:48.533684 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:48.533943 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 02:59:48.534129 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:59:48.536077 481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49d3d2ae4980489786b3b14e0bcd8f11: Bootstrap starting.
I20260925 02:59:48.538890 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:59:48.539669 481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49d3d2ae4980489786b3b14e0bcd8f11: Neither blocks nor log segments found. Creating new log.
I20260925 02:59:48.540637 481 log.cc:824] T 00000000000000000000000000000000 P 49d3d2ae4980489786b3b14e0bcd8f11: Log is configured to *not* fsync() on all Append() calls
I20260925 02:59:48.543534 481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49d3d2ae4980489786b3b14e0bcd8f11: No bootstrap required, opened a new log
I20260925 02:59:48.554557 481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49d3d2ae4980489786b3b14e0bcd8f11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ea2c62c528446181724e48a6256f9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36261 } } peers { permanent_uuid: "022251c42cdd49d28708b5c08850d24c" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } } peers { permanent_uuid: "49d3d2ae4980489786b3b14e0bcd8f11" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } }
I20260925 02:59:48.555018 481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49d3d2ae4980489786b3b14e0bcd8f11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:59:48.555158 481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49d3d2ae4980489786b3b14e0bcd8f11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d3d2ae4980489786b3b14e0bcd8f11, State: Initialized, Role: FOLLOWER
I20260925 02:59:48.555965 481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49d3d2ae4980489786b3b14e0bcd8f11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ea2c62c528446181724e48a6256f9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36261 } } peers { permanent_uuid: "022251c42cdd49d28708b5c08850d24c" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } } peers { permanent_uuid: "49d3d2ae4980489786b3b14e0bcd8f11" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } }
I20260925 02:59:48.563071 494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d3d2ae4980489786b3b14e0bcd8f11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ea2c62c528446181724e48a6256f9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36261 } } peers { permanent_uuid: "022251c42cdd49d28708b5c08850d24c" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } } peers { permanent_uuid: "49d3d2ae4980489786b3b14e0bcd8f11" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } } }
I20260925 02:59:48.563495 494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d3d2ae4980489786b3b14e0bcd8f11 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:59:48.563650 336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.543845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44314 (local address 127.18.196.254:36261)
0925 02:59:48.544054 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:48.544060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:48.545230 (+ 1170us) server_negotiation.cc:407] Connection header received
0925 02:59:48.545384 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:48.545392 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:48.545489 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:48.545601 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:48.548216 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.549294 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.551479 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.552189 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.552291 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:48.556236 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:48.556243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:48.556247 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:48.556287 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:48.560036 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.560870 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.560878 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.560883 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.561043 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.561486 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.561492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.561496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.561921 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:48.562179 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:48.563061 (+ 882us) server_negotiation.cc:299] Negotiation successful
0925 02:59:48.563290 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:59:48.565507 481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49d3d2ae4980489786b3b14e0bcd8f11 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:59:48.567467 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.543898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57228 (local address 127.18.196.252:35691)
0925 02:59:48.544093 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:48.544101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:48.547457 (+ 3356us) server_negotiation.cc:407] Connection header received
0925 02:59:48.547699 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:48.547706 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:48.547798 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:48.547891 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:48.550883 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.552447 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.555273 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.556000 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.556100 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:48.560029 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:48.560034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:48.560039 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:48.560073 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:48.563712 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.564899 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.564917 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.564919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.565119 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.565661 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.565666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.565669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.566089 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:48.566327 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:48.566897 (+ 570us) server_negotiation.cc:299] Negotiation successful
0925 02:59:48.567118 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:59:48.572691 411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.547654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48624 (local address 127.18.196.253:43901)
0925 02:59:48.547818 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:48.547823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:48.553268 (+ 5445us) server_negotiation.cc:407] Connection header received
0925 02:59:48.553526 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:48.553531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:48.553601 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:48.553667 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:48.556265 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.557645 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.560450 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.561280 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.561397 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:48.565841 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:48.565848 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:48.565852 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:48.565887 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:48.569823 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.570775 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.570785 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.570787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.570962 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.571327 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.571332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.571335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.571583 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:48.571834 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:48.572129 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 02:59:48.572351 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 02:59:48.578059 505 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49d3d2ae4980489786b3b14e0bcd8f11: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:59:48.578223 505 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:59:48.576169 414 master_runner.cc:186] Error getting master registration for 127.18.196.254:36261: OK, instance_id { permanent_uuid: "08ea2c62c528446181724e48a6256f9a" instance_seqno: 1790305187480136 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:59:48.587509 414 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:59:48.612169 333 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } attempt: 4
Sep 25 02:59:48 dist-test-slave-8zvk krb5kdc[32721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305187, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for kudu/127.18.196.252@KRBTEST.COM
I20260925 02:59:48.627281 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.612684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:35691 (local address 127.0.0.1:57236)
0925 02:59:48.612892 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.612917 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:48.613028 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:48.613394 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:48.613399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:48.613994 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:59:48.614248 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:48.614265 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.615854 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.615858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:48.616840 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:48.616852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.617848 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.617852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:48.617968 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:48.618560 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:59:48.618576 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:59:48.620600 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:59:48.625060 (+ 4460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:48.625064 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:48.625066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:48.625430 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:48.625816 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:48.625819 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:48.625821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:48.625937 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:48.626518 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:59:48.626521 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:59:48.626645 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 02:59:48.626759 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 02:59:48.627010 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 02:59:48.627540 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.612937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57236 (local address 127.18.196.252:35691)
0925 02:59:48.613092 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:48.613096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:48.613111 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:59:48.613153 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:48.613157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:48.613236 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:48.613343 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:48.614444 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.615709 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.616982 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.617711 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.617840 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:48.620770 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:48.620777 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:48.620782 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:48.620816 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:48.624917 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.625544 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.625550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.625554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.625699 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.626038 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.626044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.626049 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.626393 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:48.626550 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:48.626909 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 02:59:48.627127 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:59:48.633318 333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08ea2c62c528446181724e48a6256f9a: Bootstrap starting.
I20260925 02:59:48.637336 333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08ea2c62c528446181724e48a6256f9a: Neither blocks nor log segments found. Creating new log.
I20260925 02:59:48.638618 333 log.cc:824] T 00000000000000000000000000000000 P 08ea2c62c528446181724e48a6256f9a: Log is configured to *not* fsync() on all Append() calls
I20260925 02:59:48.641870 333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08ea2c62c528446181724e48a6256f9a: No bootstrap required, opened a new log
I20260925 02:59:48.653026 333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08ea2c62c528446181724e48a6256f9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ea2c62c528446181724e48a6256f9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36261 } } peers { permanent_uuid: "022251c42cdd49d28708b5c08850d24c" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } } peers { permanent_uuid: "49d3d2ae4980489786b3b14e0bcd8f11" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } }
I20260925 02:59:48.653432 333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08ea2c62c528446181724e48a6256f9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:59:48.653539 333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08ea2c62c528446181724e48a6256f9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ea2c62c528446181724e48a6256f9a, State: Initialized, Role: FOLLOWER
I20260925 02:59:48.654213 333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08ea2c62c528446181724e48a6256f9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ea2c62c528446181724e48a6256f9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36261 } } peers { permanent_uuid: "022251c42cdd49d28708b5c08850d24c" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } } peers { permanent_uuid: "49d3d2ae4980489786b3b14e0bcd8f11" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } }
I20260925 02:59:48.655184 508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ea2c62c528446181724e48a6256f9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ea2c62c528446181724e48a6256f9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36261 } } peers { permanent_uuid: "022251c42cdd49d28708b5c08850d24c" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } } peers { permanent_uuid: "49d3d2ae4980489786b3b14e0bcd8f11" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } } }
I20260925 02:59:48.655532 508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ea2c62c528446181724e48a6256f9a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:59:48.656488 333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08ea2c62c528446181724e48a6256f9a [sys.catalog]: configured and running, proceeding with master startup.
W20260925 02:59:48.667398 519 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08ea2c62c528446181724e48a6256f9a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:59:48.667538 519 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:59:48.666383 32734 master_runner.cc:186] Error getting master registration for 127.18.196.253:43901: OK, instance_id { permanent_uuid: "022251c42cdd49d28708b5c08850d24c" instance_seqno: 1790305187975027 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:59:48.678908 32734 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:59:48.701632 407 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } attempt: 5
W20260925 02:59:48.702493 435 tablet.cc:2369] T 00000000000000000000000000000000 P 49d3d2ae4980489786b3b14e0bcd8f11: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 02:59:48 dist-test-slave-8zvk krb5kdc[32721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305187, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.253@KRBTEST.COM for kudu/127.18.196.252@KRBTEST.COM
I20260925 02:59:48.716226 409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.702271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:35691 (local address 127.0.0.1:57242)
0925 02:59:48.702549 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:48.702570 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:48.702690 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:48.703481 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:48.703484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:48.703873 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:59:48.704024 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:48.704032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.705058 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.705063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:48.705935 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:48.705947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.706687 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.706692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:48.706973 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:48.707594 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:59:48.707621 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:59:48.709980 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:59:48.713752 (+ 3772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:48.713758 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:48.713762 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:48.714117 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:48.714501 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:48.714507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:48.714511 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:48.714642 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:48.715224 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:59:48.715229 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:59:48.715399 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 02:59:48.715551 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 02:59:48.715845 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260925 02:59:48.716251 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:48.702341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57242 (local address 127.18.196.252:35691)
0925 02:59:48.702959 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:48.702965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:48.702984 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:59:48.703046 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:48.703050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:48.703124 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:48.703220 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:48.704176 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.704911 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.706092 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:48.706536 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:48.706670 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:48.710236 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:48.710242 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:48.710247 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:48.710298 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:48.713621 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.714237 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.714244 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.714248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.714389 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:48.714756 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:48.714762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:48.714766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:48.715100 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:48.715243 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:48.715663 (+ 420us) server_negotiation.cc:299] Negotiation successful
0925 02:59:48.715877 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461}
W20260925 02:59:48.719800 32755 tablet.cc:2369] T 00000000000000000000000000000000 P 08ea2c62c528446181724e48a6256f9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:59:48.720829 407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c: Bootstrap starting.
I20260925 02:59:48.723841 407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c: Neither blocks nor log segments found. Creating new log.
I20260925 02:59:48.724797 407 log.cc:824] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:59:48.728744 407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c: No bootstrap required, opened a new log
I20260925 02:59:48.736418 407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ea2c62c528446181724e48a6256f9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36261 } } peers { permanent_uuid: "022251c42cdd49d28708b5c08850d24c" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } } peers { permanent_uuid: "49d3d2ae4980489786b3b14e0bcd8f11" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } }
I20260925 02:59:48.736737 407 raft_consensus.cc:374] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:59:48.736881 407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022251c42cdd49d28708b5c08850d24c, State: Initialized, Role: FOLLOWER
I20260925 02:59:48.737445 407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ea2c62c528446181724e48a6256f9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36261 } } peers { permanent_uuid: "022251c42cdd49d28708b5c08850d24c" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } } peers { permanent_uuid: "49d3d2ae4980489786b3b14e0bcd8f11" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } }
I20260925 02:59:48.738313 522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ea2c62c528446181724e48a6256f9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36261 } } peers { permanent_uuid: "022251c42cdd49d28708b5c08850d24c" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } } peers { permanent_uuid: "49d3d2ae4980489786b3b14e0bcd8f11" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } } }
I20260925 02:59:48.738718 522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:59:48.739491 407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:59:48.750690 522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:59:48.751056 522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ea2c62c528446181724e48a6256f9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36261 } } peers { permanent_uuid: "022251c42cdd49d28708b5c08850d24c" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } } peers { permanent_uuid: "49d3d2ae4980489786b3b14e0bcd8f11" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } }
I20260925 02:59:48.752830 522 leader_election.cc:288] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08ea2c62c528446181724e48a6256f9a (127.18.196.254:36261), 49d3d2ae4980489786b3b14e0bcd8f11 (127.18.196.252:35691)
I20260925 02:59:48.754160 308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "022251c42cdd49d28708b5c08850d24c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ea2c62c528446181724e48a6256f9a" is_pre_election: true
I20260925 02:59:48.754812 308 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 08ea2c62c528446181724e48a6256f9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 022251c42cdd49d28708b5c08850d24c in term 0.
I20260925 02:59:48.755851 358 leader_election.cc:302] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022251c42cdd49d28708b5c08850d24c, 08ea2c62c528446181724e48a6256f9a; no voters:
I20260925 02:59:48.755760 456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "022251c42cdd49d28708b5c08850d24c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d3d2ae4980489786b3b14e0bcd8f11" is_pre_election: true
I20260925 02:59:48.756340 522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:59:48.756340 456 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 49d3d2ae4980489786b3b14e0bcd8f11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 022251c42cdd49d28708b5c08850d24c in term 0.
I20260925 02:59:48.756475 522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:59:48.756565 522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:59:48.761355 522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ea2c62c528446181724e48a6256f9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36261 } } peers { permanent_uuid: "022251c42cdd49d28708b5c08850d24c" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } } peers { permanent_uuid: "49d3d2ae4980489786b3b14e0bcd8f11" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } }
I20260925 02:59:48.762728 308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "022251c42cdd49d28708b5c08850d24c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ea2c62c528446181724e48a6256f9a"
I20260925 02:59:48.763034 308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08ea2c62c528446181724e48a6256f9a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:59:48.767524 308 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 08ea2c62c528446181724e48a6256f9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 022251c42cdd49d28708b5c08850d24c in term 1.
I20260925 02:59:48.768168 358 leader_election.cc:302] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022251c42cdd49d28708b5c08850d24c, 08ea2c62c528446181724e48a6256f9a; no voters:
I20260925 02:59:48.777006 456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "022251c42cdd49d28708b5c08850d24c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d3d2ae4980489786b3b14e0bcd8f11"
I20260925 02:59:48.777289 456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49d3d2ae4980489786b3b14e0bcd8f11 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:59:48.782195 456 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 49d3d2ae4980489786b3b14e0bcd8f11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 022251c42cdd49d28708b5c08850d24c in term 1.
I20260925 02:59:48.783246 522 leader_election.cc:288] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [CANDIDATE]: Term 1 election: Requested vote from peers 08ea2c62c528446181724e48a6256f9a (127.18.196.254:36261), 49d3d2ae4980489786b3b14e0bcd8f11 (127.18.196.252:35691)
I20260925 02:59:48.783761 522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:59:48.784265 522 raft_consensus.cc:686] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [term 1 LEADER]: Becoming Leader. State: Replica: 022251c42cdd49d28708b5c08850d24c, State: Running, Role: LEADER
I20260925 02:59:48.785422 522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ea2c62c528446181724e48a6256f9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36261 } } peers { permanent_uuid: "022251c42cdd49d28708b5c08850d24c" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } } peers { permanent_uuid: "49d3d2ae4980489786b3b14e0bcd8f11" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } }
I20260925 02:59:48.801622 528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 022251c42cdd49d28708b5c08850d24c. Latest consensus state: current_term: 1 leader_uuid: "022251c42cdd49d28708b5c08850d24c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ea2c62c528446181724e48a6256f9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36261 } } peers { permanent_uuid: "022251c42cdd49d28708b5c08850d24c" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } } peers { permanent_uuid: "49d3d2ae4980489786b3b14e0bcd8f11" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } } }
I20260925 02:59:48.802209 528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [sys.catalog]: This master's current role is: LEADER
I20260925 02:59:48.806641 530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:59:48.825500 530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:59:48.842968 537 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 02:59:48.851781 456 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 49d3d2ae4980489786b3b14e0bcd8f11 [term 1 FOLLOWER]: Refusing update from remote peer 022251c42cdd49d28708b5c08850d24c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:59:48.852696 522 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [LEADER]: Connected to new peer: Peer: permanent_uuid: "49d3d2ae4980489786b3b14e0bcd8f11" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:59:48.854084 308 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08ea2c62c528446181724e48a6256f9a [term 1 FOLLOWER]: Refusing update from remote peer 022251c42cdd49d28708b5c08850d24c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:59:48.858201 522 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [LEADER]: Connected to new peer: Peer: permanent_uuid: "08ea2c62c528446181724e48a6256f9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:59:48.865772 494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d3d2ae4980489786b3b14e0bcd8f11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 022251c42cdd49d28708b5c08850d24c. Latest consensus state: current_term: 1 leader_uuid: "022251c42cdd49d28708b5c08850d24c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ea2c62c528446181724e48a6256f9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36261 } } peers { permanent_uuid: "022251c42cdd49d28708b5c08850d24c" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } } peers { permanent_uuid: "49d3d2ae4980489786b3b14e0bcd8f11" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } } }
I20260925 02:59:48.866065 494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d3d2ae4980489786b3b14e0bcd8f11 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:59:48.874296 508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ea2c62c528446181724e48a6256f9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 022251c42cdd49d28708b5c08850d24c. Latest consensus state: current_term: 1 leader_uuid: "022251c42cdd49d28708b5c08850d24c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ea2c62c528446181724e48a6256f9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36261 } } peers { permanent_uuid: "022251c42cdd49d28708b5c08850d24c" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } } peers { permanent_uuid: "49d3d2ae4980489786b3b14e0bcd8f11" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } } }
I20260925 02:59:48.874548 508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ea2c62c528446181724e48a6256f9a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:59:48.886474 494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d3d2ae4980489786b3b14e0bcd8f11 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "022251c42cdd49d28708b5c08850d24c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ea2c62c528446181724e48a6256f9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36261 } } peers { permanent_uuid: "022251c42cdd49d28708b5c08850d24c" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } } peers { permanent_uuid: "49d3d2ae4980489786b3b14e0bcd8f11" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } } }
I20260925 02:59:48.886807 494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d3d2ae4980489786b3b14e0bcd8f11 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:59:48.896689 508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ea2c62c528446181724e48a6256f9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "022251c42cdd49d28708b5c08850d24c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ea2c62c528446181724e48a6256f9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36261 } } peers { permanent_uuid: "022251c42cdd49d28708b5c08850d24c" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } } peers { permanent_uuid: "49d3d2ae4980489786b3b14e0bcd8f11" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } } }
I20260925 02:59:48.896942 508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ea2c62c528446181724e48a6256f9a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:59:48.901868 530 catalog_manager.cc:1347] Generated new cluster ID: c151574cce99448a835640cef3d97e80
I20260925 02:59:48.901973 530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:59:48.902582 522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "022251c42cdd49d28708b5c08850d24c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ea2c62c528446181724e48a6256f9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36261 } } peers { permanent_uuid: "022251c42cdd49d28708b5c08850d24c" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } } peers { permanent_uuid: "49d3d2ae4980489786b3b14e0bcd8f11" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } } }
I20260925 02:59:48.902949 522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [sys.catalog]: This master's current role is: LEADER
I20260925 02:59:48.903250 522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "022251c42cdd49d28708b5c08850d24c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ea2c62c528446181724e48a6256f9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36261 } } peers { permanent_uuid: "022251c42cdd49d28708b5c08850d24c" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 43901 } } peers { permanent_uuid: "49d3d2ae4980489786b3b14e0bcd8f11" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 35691 } } }
I20260925 02:59:48.903522 522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c [sys.catalog]: This master's current role is: LEADER
I20260925 02:59:48.919970 530 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:59:48.921975 530 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:59:48.956462 530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 022251c42cdd49d28708b5c08850d24c: Generated new TSK 0
I20260925 02:59:48.957413 530 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:59:49.580310 505 catalog_manager.cc:1259] Loaded cluster ID: c151574cce99448a835640cef3d97e80
I20260925 02:59:49.580400 505 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 49d3d2ae4980489786b3b14e0bcd8f11: loading cluster ID for follower catalog manager: success
I20260925 02:59:49.582764 505 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 49d3d2ae4980489786b3b14e0bcd8f11: acquiring CA information for follower catalog manager: success
I20260925 02:59:49.584487 505 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 49d3d2ae4980489786b3b14e0bcd8f11: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:59:49.669515 519 catalog_manager.cc:1259] Loaded cluster ID: c151574cce99448a835640cef3d97e80
I20260925 02:59:49.669658 519 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 08ea2c62c528446181724e48a6256f9a: loading cluster ID for follower catalog manager: success
I20260925 02:59:49.672405 519 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 08ea2c62c528446181724e48a6256f9a: acquiring CA information for follower catalog manager: success
I20260925 02:59:49.674100 519 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 08ea2c62c528446181724e48a6256f9a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:59:50.095829 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 32734
I20260925 02:59:50.097146 328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:59:50.206879 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 32734
I20260925 02:59:50.227627 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 340
I20260925 02:59:50.228639 402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:59:50.335063 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 340
W20260925 02:59:50.356503 433 connection.cc:535] server connection from 127.0.0.1:57242 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:59:50.356871 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 414
I20260925 02:59:50.357981 476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:59:50.462118 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 414
2026-09-25T02:59:50Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4720 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:59:50 dist-test-slave-8zvk krb5kdc[556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:59:50 dist-test-slave-8zvk krb5kdc[556](info): set up 2 sockets
Sep 25 02:59:50 dist-test-slave-8zvk krb5kdc[556](info): commencing operation
krb5kdc: starting...
W20260925 02:59:51.550443 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.006s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 02:59:51 dist-test-slave-8zvk krb5kdc[556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:59:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:59:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 02:59:51.768491 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:45615
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:45215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:45615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:59:52.091753 569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:59:52.092077 569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:59:52.092239 569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:59:52.099285 569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:59:52.099426 569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:59:52.099524 569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:59:52.099618 569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:59:52.099697 569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:59:52.109046 569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:45215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:45615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:45615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:59:52.110325 569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:59:52.111625 569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:59:52.122745 575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:52.122866 574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:52.122821 577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:59:52.123956 569 server_base.cc:1034] running on GCE node
I20260925 02:59:52.124614 569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:59:52.125830 569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:59:52.127023 569 hybrid_clock.cc:648] HybridClock initialized: now 1790305192126979 us; error 29 us; skew 500 ppm
I20260925 02:59:52.127312 569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:59:52 dist-test-slave-8zvk krb5kdc[556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305192, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:59:52.130163 569 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 02:59:52.131639 569 webserver.cc:460] Webserver started at http://127.18.196.254:35981/ using document root <none> and password file <none>
I20260925 02:59:52.132321 569 fs_manager.cc:359] Metadata directory not provided
I20260925 02:59:52.132449 569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:59:52.132783 569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:59:52.135696 569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "c1d4f6475bce4e5589d1cfe1fa758b0b"
format_stamp: "Formatted at 2026-09-25 02:59:52 on dist-test-slave-8zvk"
server_key: "5113bac7ac53fe255e37fd0246b03186"
server_key_iv: "042cd9d8e5625bbe484f2a7c5c02f0d9"
server_key_version: "encryptionkey@0"
I20260925 02:59:52.136425 569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "c1d4f6475bce4e5589d1cfe1fa758b0b"
format_stamp: "Formatted at 2026-09-25 02:59:52 on dist-test-slave-8zvk"
server_key: "5113bac7ac53fe255e37fd0246b03186"
server_key_iv: "042cd9d8e5625bbe484f2a7c5c02f0d9"
server_key_version: "encryptionkey@0"
I20260925 02:59:52.141247 569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:59:52.144574 584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:52.145781 569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:59:52.145961 569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "c1d4f6475bce4e5589d1cfe1fa758b0b"
format_stamp: "Formatted at 2026-09-25 02:59:52 on dist-test-slave-8zvk"
server_key: "5113bac7ac53fe255e37fd0246b03186"
server_key_iv: "042cd9d8e5625bbe484f2a7c5c02f0d9"
server_key_version: "encryptionkey@0"
I20260925 02:59:52.146152 569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:59:52.164243 569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:59:52.167847 569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:59:52.168116 569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:59:52.185813 569 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:45615
I20260925 02:59:52.185845 635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:45615 every 8 connection(s)
I20260925 02:59:52.187115 569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:59:52.190277 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 569
I20260925 02:59:52.190511 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:59:52.191273 19219 external_mini_cluster.cc:1428] Setting key 7b3990ed8679d40f741dd7286c9a1bac
I20260925 02:59:52.191612 636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:59:52.205740 636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1d4f6475bce4e5589d1cfe1fa758b0b: Bootstrap starting.
Sep 25 02:59:52 dist-test-slave-8zvk krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:59:52.209348 636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1d4f6475bce4e5589d1cfe1fa758b0b: Neither blocks nor log segments found. Creating new log.
I20260925 02:59:52.210652 636 log.cc:824] T 00000000000000000000000000000000 P c1d4f6475bce4e5589d1cfe1fa758b0b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:59:52.213850 636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1d4f6475bce4e5589d1cfe1fa758b0b: No bootstrap required, opened a new log
I20260925 02:59:52.215466 639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:52.193033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53192 (local address 127.18.196.254:45615)
0925 02:59:52.193864 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:52.193875 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:52.193913 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:59:52.196144 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:52.196163 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:52.196501 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:52.196798 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:52.198575 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:52.199841 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:52.201619 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:52.202305 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:52.202431 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:52.207319 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:52.207329 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:52.207353 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:52.207400 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:52.211047 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:52.211983 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:52.211992 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:52.212000 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:52.212202 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:52.212558 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:52.212562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:52.212565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:52.213211 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:52.213476 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:52.213833 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 02:59:52.214232 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":358,"threads_started":1}
I20260925 02:59:52.219157 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:59:52.222415 636 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1d4f6475bce4e5589d1cfe1fa758b0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d4f6475bce4e5589d1cfe1fa758b0b" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 45615 } }
I20260925 02:59:52.222774 636 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1d4f6475bce4e5589d1cfe1fa758b0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:59:52.222916 636 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1d4f6475bce4e5589d1cfe1fa758b0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d4f6475bce4e5589d1cfe1fa758b0b, State: Initialized, Role: FOLLOWER
I20260925 02:59:52.223511 636 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1d4f6475bce4e5589d1cfe1fa758b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d4f6475bce4e5589d1cfe1fa758b0b" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 45615 } }
I20260925 02:59:52.223744 636 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1d4f6475bce4e5589d1cfe1fa758b0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:59:52.223902 636 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1d4f6475bce4e5589d1cfe1fa758b0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:59:52.224072 636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1d4f6475bce4e5589d1cfe1fa758b0b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:59:52.226135 636 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1d4f6475bce4e5589d1cfe1fa758b0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d4f6475bce4e5589d1cfe1fa758b0b" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 45615 } }
I20260925 02:59:52.226615 636 leader_election.cc:302] T 00000000000000000000000000000000 P c1d4f6475bce4e5589d1cfe1fa758b0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1d4f6475bce4e5589d1cfe1fa758b0b; no voters:
I20260925 02:59:52.227195 636 leader_election.cc:288] T 00000000000000000000000000000000 P c1d4f6475bce4e5589d1cfe1fa758b0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:59:52.227315 641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1d4f6475bce4e5589d1cfe1fa758b0b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:59:52.228008 641 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1d4f6475bce4e5589d1cfe1fa758b0b [term 1 LEADER]: Becoming Leader. State: Replica: c1d4f6475bce4e5589d1cfe1fa758b0b, State: Running, Role: LEADER
I20260925 02:59:52.228570 641 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1d4f6475bce4e5589d1cfe1fa758b0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d4f6475bce4e5589d1cfe1fa758b0b" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 45615 } }
I20260925 02:59:52.229182 636 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1d4f6475bce4e5589d1cfe1fa758b0b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:59:52.231560 643 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1d4f6475bce4e5589d1cfe1fa758b0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1d4f6475bce4e5589d1cfe1fa758b0b. Latest consensus state: current_term: 1 leader_uuid: "c1d4f6475bce4e5589d1cfe1fa758b0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d4f6475bce4e5589d1cfe1fa758b0b" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 45615 } } }
I20260925 02:59:52.231923 643 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1d4f6475bce4e5589d1cfe1fa758b0b [sys.catalog]: This master's current role is: LEADER
I20260925 02:59:52.231660 642 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1d4f6475bce4e5589d1cfe1fa758b0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1d4f6475bce4e5589d1cfe1fa758b0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d4f6475bce4e5589d1cfe1fa758b0b" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 45615 } } }
I20260925 02:59:52.232043 642 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1d4f6475bce4e5589d1cfe1fa758b0b [sys.catalog]: This master's current role is: LEADER
I20260925 02:59:52.234383 647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:59:52.238595 647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:59:52.246273 647 catalog_manager.cc:1347] Generated new cluster ID: 3004bff6fa914a08b91e713ddd2117d5
I20260925 02:59:52.246381 647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:59:52.256208 647 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:59:52.257383 647 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:59:52.263862 647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1d4f6475bce4e5589d1cfe1fa758b0b: Generated new TSK 0
I20260925 02:59:52.264509 647 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:59:52.279175 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 569
I20260925 02:59:52.280265 631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:59:52.369940 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 569
2026-09-25T02:59:52Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1904 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:59:52 dist-test-slave-8zvk krb5kdc[664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:59:52 dist-test-slave-8zvk krb5kdc[664](info): set up 2 sockets
Sep 25 02:59:52 dist-test-slave-8zvk krb5kdc[664](info): commencing operation
krb5kdc: starting...
W20260925 02:59:53.442374 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 02:59:53 dist-test-slave-8zvk krb5kdc[664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:59:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:59:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 02:59:53.660322 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:39615
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:37687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:39615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:59:53.966310 677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:59:53.966579 677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:59:53.966725 677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:59:53.973470 677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:59:53.973582 677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:59:53.973658 677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:59:53.973753 677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:59:53.973812 677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:59:53.983074 677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:37687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:39615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:39615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:59:53.984203 677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:59:53.985405 677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:59:53.996543 685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:53.996573 683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:53.997133 682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:59:53.997815 677 server_base.cc:1034] running on GCE node
I20260925 02:59:53.998481 677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:59:53.999898 677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:59:54.001098 677 hybrid_clock.cc:648] HybridClock initialized: now 1790305194001060 us; error 32 us; skew 500 ppm
I20260925 02:59:54.001430 677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:59:54 dist-test-slave-8zvk krb5kdc[664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305194, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:59:54.009835 677 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 02:59:54.011319 677 webserver.cc:460] Webserver started at http://127.18.196.254:46265/ using document root <none> and password file <none>
I20260925 02:59:54.011970 677 fs_manager.cc:359] Metadata directory not provided
I20260925 02:59:54.012118 677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:59:54.012428 677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:59:54.015328 677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "fb2916c744e440ec97ee79298c2eed96"
format_stamp: "Formatted at 2026-09-25 02:59:54 on dist-test-slave-8zvk"
server_key: "94a7defae33de466c27aae988bc5bbbc"
server_key_iv: "c6c94a417f02982cb0a44c2fab4f0bff"
server_key_version: "encryptionkey@0"
I20260925 02:59:54.016073 677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "fb2916c744e440ec97ee79298c2eed96"
format_stamp: "Formatted at 2026-09-25 02:59:54 on dist-test-slave-8zvk"
server_key: "94a7defae33de466c27aae988bc5bbbc"
server_key_iv: "c6c94a417f02982cb0a44c2fab4f0bff"
server_key_version: "encryptionkey@0"
I20260925 02:59:54.020928 677 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:59:54.024070 692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:54.025322 677 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260925 02:59:54.025496 677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "fb2916c744e440ec97ee79298c2eed96"
format_stamp: "Formatted at 2026-09-25 02:59:54 on dist-test-slave-8zvk"
server_key: "94a7defae33de466c27aae988bc5bbbc"
server_key_iv: "c6c94a417f02982cb0a44c2fab4f0bff"
server_key_version: "encryptionkey@0"
I20260925 02:59:54.025684 677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:59:54.039376 677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:59:54.042927 677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:59:54.043195 677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:59:54.061098 677 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:39615
I20260925 02:59:54.061141 743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:39615 every 8 connection(s)
I20260925 02:59:54.062371 677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:59:54.065627 744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:59:54.071051 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 677
I20260925 02:59:54.071257 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 02:59:54.071852 19219 external_mini_cluster.cc:1428] Setting key be8df4d0c917ce4ce85084b2a1ef9196
I20260925 02:59:54.075176 744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96: Bootstrap starting.
I20260925 02:59:54.078385 744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96: Neither blocks nor log segments found. Creating new log.
I20260925 02:59:54.079536 744 log.cc:824] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96: Log is configured to *not* fsync() on all Append() calls
I20260925 02:59:54.082104 744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96: No bootstrap required, opened a new log
Sep 25 02:59:54 dist-test-slave-8zvk krb5kdc[664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:59:54.090595 744 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2916c744e440ec97ee79298c2eed96" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39615 } }
I20260925 02:59:54.091050 744 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:59:54.091156 744 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb2916c744e440ec97ee79298c2eed96, State: Initialized, Role: FOLLOWER
I20260925 02:59:54.091773 744 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2916c744e440ec97ee79298c2eed96" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39615 } }
I20260925 02:59:54.092056 744 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:59:54.092195 744 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:59:54.092404 744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:59:54.093933 747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:54.073680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41500 (local address 127.18.196.254:39615)
0925 02:59:54.074433 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:54.074444 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:54.074479 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:59:54.076190 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:54.076214 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:54.076558 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:54.076858 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:54.078707 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:54.080144 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:54.081518 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:54.082220 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:54.082432 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:54.086726 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:54.086733 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:54.086745 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:54.086778 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:54.089758 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:54.090641 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:54.090648 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:54.090653 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:54.090800 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:54.091281 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:54.091285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:54.091288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:54.091826 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:54.092118 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:54.092578 (+ 460us) server_negotiation.cc:299] Negotiation successful
0925 02:59:54.092956 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":354,"threads_started":1}
I20260925 02:59:54.094775 744 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2916c744e440ec97ee79298c2eed96" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39615 } }
I20260925 02:59:54.095329 744 leader_election.cc:302] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb2916c744e440ec97ee79298c2eed96; no voters:
I20260925 02:59:54.095896 744 leader_election.cc:288] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:59:54.096028 749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:59:54.097370 749 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 1 LEADER]: Becoming Leader. State: Replica: fb2916c744e440ec97ee79298c2eed96, State: Running, Role: LEADER
I20260925 02:59:54.097859 749 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2916c744e440ec97ee79298c2eed96" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39615 } }
I20260925 02:59:54.097956 744 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:59:54.101214 751 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb2916c744e440ec97ee79298c2eed96. Latest consensus state: current_term: 1 leader_uuid: "fb2916c744e440ec97ee79298c2eed96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2916c744e440ec97ee79298c2eed96" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39615 } } }
I20260925 02:59:54.101536 751 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [sys.catalog]: This master's current role is: LEADER
I20260925 02:59:54.102041 750 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb2916c744e440ec97ee79298c2eed96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2916c744e440ec97ee79298c2eed96" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39615 } } }
I20260925 02:59:54.102422 750 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [sys.catalog]: This master's current role is: LEADER
I20260925 02:59:54.108021 756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:59:54.113618 756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:59:54.121764 756 catalog_manager.cc:1347] Generated new cluster ID: e0ba09d6c9704269960ffab4fa1c5745
I20260925 02:59:54.121870 756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:59:54.155537 756 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:59:54.156836 756 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:59:54.162120 756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96: Generated new TSK 0
I20260925 02:59:54.162899 756 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
I20260925 02:59:54.226222 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:39615
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:37687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:59:54.551121 770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:59:54.551393 770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:59:54.551589 770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:59:54.558686 770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:59:54.558794 770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:59:54.559000 770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 02:59:54.568642 770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:37687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:39615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:59:54.569916 770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:59:54.571270 770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:59:54.584033 778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:54.584095 776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:54.584117 775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:59:54.585449 770 server_base.cc:1034] running on GCE node
I20260925 02:59:54.586063 770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:59:54.587062 770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:59:54.588289 770 hybrid_clock.cc:648] HybridClock initialized: now 1790305194588253 us; error 47 us; skew 500 ppm
I20260925 02:59:54.588603 770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:59:54 dist-test-slave-8zvk krb5kdc[664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305194, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:59:54.591653 770 init.cc:377] Logged in from keytab as kudu/127.18.196.193@KRBTEST.COM (short username kudu)
I20260925 02:59:54.593173 770 webserver.cc:460] Webserver started at http://127.18.196.193:40759/ using document root <none> and password file <none>
I20260925 02:59:54.593855 770 fs_manager.cc:359] Metadata directory not provided
I20260925 02:59:54.594002 770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:59:54.594310 770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:59:54.597327 770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "e83773d3745a43149e1afdf48c90e0b1"
format_stamp: "Formatted at 2026-09-25 02:59:54 on dist-test-slave-8zvk"
server_key: "7a4cf11b744cb2c6c3f4e0028439c259"
server_key_iv: "49b8041bf5ddbd544c14ba973d85c19f"
server_key_version: "encryptionkey@0"
I20260925 02:59:54.598099 770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "e83773d3745a43149e1afdf48c90e0b1"
format_stamp: "Formatted at 2026-09-25 02:59:54 on dist-test-slave-8zvk"
server_key: "7a4cf11b744cb2c6c3f4e0028439c259"
server_key_iv: "49b8041bf5ddbd544c14ba973d85c19f"
server_key_version: "encryptionkey@0"
I20260925 02:59:54.603282 770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 02:59:54.606504 785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:54.607772 770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:59:54.607954 770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "e83773d3745a43149e1afdf48c90e0b1"
format_stamp: "Formatted at 2026-09-25 02:59:54 on dist-test-slave-8zvk"
server_key: "7a4cf11b744cb2c6c3f4e0028439c259"
server_key_iv: "49b8041bf5ddbd544c14ba973d85c19f"
server_key_version: "encryptionkey@0"
I20260925 02:59:54.608147 770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:59:54.632745 770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:59:54.636787 770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:59:54.637089 770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:59:54.638056 770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:59:54.639459 770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:59:54.639560 770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:54.639710 770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:59:54.639791 770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:54.672791 770 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:32845
I20260925 02:59:54.672878 898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:32845 every 8 connection(s)
I20260925 02:59:54.674074 770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 02:59:54.682253 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 770
I20260925 02:59:54.682479 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 02:59:54.683178 19219 external_mini_cluster.cc:1428] Setting key 5066db315e6698ece9deca28ae13e873
Sep 25 02:59:54 dist-test-slave-8zvk krb5kdc[664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305194, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:59:54.695068 901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:54.676523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:60593 (local address 127.18.196.254:39615)
0925 02:59:54.676967 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:54.676971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:54.677730 (+ 759us) server_negotiation.cc:407] Connection header received
0925 02:59:54.679590 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:54.679596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:54.679698 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:54.679823 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:54.681236 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:54.682126 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:54.684023 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:54.684494 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:54.684595 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:54.688183 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:54.688192 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:54.688200 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:54.688242 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:54.691673 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:54.692478 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:54.692486 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:54.692492 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:54.692643 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:54.693111 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:54.693119 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:54.693125 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:54.693526 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:54.693705 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:54.694559 (+ 854us) server_negotiation.cc:299] Negotiation successful
0925 02:59:54.694749 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":336,"thread_start_us":257,"threads_started":1}
I20260925 02:59:54.696432 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:54.676911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.193:60593)
0925 02:59:54.677578 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:54.677620 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:54.679327 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:54.680049 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:54.680057 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:54.680527 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:59:54.681066 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:54.681083 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:54.682313 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:54.682317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:54.683868 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:54.683887 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:54.684739 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:54.684748 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:54.685077 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:54.685814 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:59:54.685836 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:59:54.687962 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:59:54.691810 (+ 3848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:54.691816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:54.691829 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:54.692290 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:54.692759 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:54.692762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:54.692765 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:54.692947 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:54.693683 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:59:54.693692 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:59:54.694134 (+ 442us) client_negotiation.cc:769] Sending connection context
0925 02:59:54.694441 (+ 307us) client_negotiation.cc:240] Negotiation successful
0925 02:59:54.695046 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":260,"threads_started":1}
I20260925 02:59:54.697969 899 heartbeater.cc:347] Connected to a master server at 127.18.196.254:39615
I20260925 02:59:54.698331 899 heartbeater.cc:464] Registering TS with master...
I20260925 02:59:54.699137 899 heartbeater.cc:511] Master 127.18.196.254:39615 requested a full tablet report, sending...
I20260925 02:59:54.701244 709 ts_manager.cc:194] Registered new tserver with Master: e83773d3745a43149e1afdf48c90e0b1 (127.18.196.193:32845)
I20260925 02:59:54.702679 709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.193@KRBTEST.COM'} at 127.18.196.193:60593
WARNING: no policy specified for kudu/127.18.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
I20260925 02:59:54.738070 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:39615
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:37687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:59:55.054279 906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:59:55.054539 906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:59:55.054702 906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:59:55.061904 906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:59:55.062000 906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:59:55.062165 906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 02:59:55.071424 906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:37687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:39615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:59:55.072523 906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:59:55.073737 906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:59:55.086138 911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:55.086094 912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:55.086109 914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:59:55.086942 906 server_base.cc:1034] running on GCE node
I20260925 02:59:55.087553 906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:59:55.088435 906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:59:55.089655 906 hybrid_clock.cc:648] HybridClock initialized: now 1790305195089623 us; error 43 us; skew 500 ppm
I20260925 02:59:55.089963 906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:59:55 dist-test-slave-8zvk krb5kdc[664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305195, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:59:55.092958 906 init.cc:377] Logged in from keytab as kudu/127.18.196.194@KRBTEST.COM (short username kudu)
I20260925 02:59:55.094303 906 webserver.cc:460] Webserver started at http://127.18.196.194:35843/ using document root <none> and password file <none>
I20260925 02:59:55.094992 906 fs_manager.cc:359] Metadata directory not provided
I20260925 02:59:55.095132 906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:59:55.095413 906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:59:55.098716 906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "621383aa8000491bbdc36bf8de3e80ba"
format_stamp: "Formatted at 2026-09-25 02:59:55 on dist-test-slave-8zvk"
server_key: "23bd3733b288b8508b3932600d3bcd0c"
server_key_iv: "57e29a9683bad97027e504a57c9a8e06"
server_key_version: "encryptionkey@0"
I20260925 02:59:55.099751 906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "621383aa8000491bbdc36bf8de3e80ba"
format_stamp: "Formatted at 2026-09-25 02:59:55 on dist-test-slave-8zvk"
server_key: "23bd3733b288b8508b3932600d3bcd0c"
server_key_iv: "57e29a9683bad97027e504a57c9a8e06"
server_key_version: "encryptionkey@0"
I20260925 02:59:55.104641 906 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:59:55.107928 921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:55.109282 906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:59:55.109445 906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "621383aa8000491bbdc36bf8de3e80ba"
format_stamp: "Formatted at 2026-09-25 02:59:55 on dist-test-slave-8zvk"
server_key: "23bd3733b288b8508b3932600d3bcd0c"
server_key_iv: "57e29a9683bad97027e504a57c9a8e06"
server_key_version: "encryptionkey@0"
I20260925 02:59:55.109686 906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:59:55.129558 906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:59:55.133179 906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:59:55.133446 906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:59:55.134285 906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:59:55.135531 906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:59:55.135643 906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:55.135787 906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:59:55.135857 906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:55.167840 906 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:39683
I20260925 02:59:55.167920 1033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:39683 every 8 connection(s)
I20260925 02:59:55.169060 906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 02:59:55.178515 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 906
I20260925 02:59:55.178776 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 02:59:55.179312 19219 external_mini_cluster.cc:1428] Setting key 09971d1998a2927aa113184a2711e726
Sep 25 02:59:55 dist-test-slave-8zvk krb5kdc[664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305195, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:59:55.187260 901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:55.171238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:44777 (local address 127.18.196.254:39615)
0925 02:59:55.171432 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:55.171438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:55.172468 (+ 1030us) server_negotiation.cc:407] Connection header received
0925 02:59:55.174080 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:55.174084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:55.174147 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:55.174231 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:55.175598 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:55.176220 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:55.177570 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:55.177958 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:55.178062 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:55.181658 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:55.181663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:55.181667 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:55.181697 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:55.184248 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:55.185149 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:55.185153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:55.185155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:55.185269 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:55.185725 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:55.185727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:55.185729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:55.185965 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:55.186067 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:55.186805 (+ 738us) server_negotiation.cc:299] Negotiation successful
0925 02:59:55.187030 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:59:55.188593 1036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:55.171689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.194:44777)
0925 02:59:55.172335 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:55.172367 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:55.173851 (+ 1484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:55.174421 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:55.174429 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:55.174977 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:59:55.175465 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:55.175480 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:55.176341 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:55.176344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:55.177454 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:55.177466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:55.178155 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:55.178160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:55.178350 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:55.179037 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:59:55.179063 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:59:55.181427 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:59:55.184421 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:55.184432 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:55.184447 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:55.185040 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:55.185375 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:55.185383 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:55.185390 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:55.185622 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:55.186084 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:59:55.186093 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:59:55.186472 (+ 379us) client_negotiation.cc:769] Sending connection context
0925 02:59:55.186729 (+ 257us) client_negotiation.cc:240] Negotiation successful
0925 02:59:55.187252 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":268,"threads_started":1}
I20260925 02:59:55.189925 1034 heartbeater.cc:347] Connected to a master server at 127.18.196.254:39615
I20260925 02:59:55.190225 1034 heartbeater.cc:464] Registering TS with master...
I20260925 02:59:55.191069 1034 heartbeater.cc:511] Master 127.18.196.254:39615 requested a full tablet report, sending...
I20260925 02:59:55.192497 709 ts_manager.cc:194] Registered new tserver with Master: 621383aa8000491bbdc36bf8de3e80ba (127.18.196.194:39683)
I20260925 02:59:55.193434 709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.194@KRBTEST.COM'} at 127.18.196.194:44777
WARNING: no policy specified for kudu/127.18.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
I20260925 02:59:55.234522 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:39615
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:37687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:59:55.545571 1039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:59:55.545831 1039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:59:55.546025 1039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:59:55.553363 1039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:59:55.553464 1039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:59:55.553730 1039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 02:59:55.563244 1039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:37687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:39615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:59:55.564491 1039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:59:55.565778 1039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:59:55.578851 1044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:55.578760 1047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:55.578809 1045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:59:55.579524 1039 server_base.cc:1034] running on GCE node
I20260925 02:59:55.580133 1039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:59:55.581000 1039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:59:55.582221 1039 hybrid_clock.cc:648] HybridClock initialized: now 1790305195582179 us; error 56 us; skew 500 ppm
I20260925 02:59:55.582523 1039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:59:55 dist-test-slave-8zvk krb5kdc[664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305195, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:59:55.585520 1039 init.cc:377] Logged in from keytab as kudu/127.18.196.195@KRBTEST.COM (short username kudu)
I20260925 02:59:55.586802 1039 webserver.cc:460] Webserver started at http://127.18.196.195:43035/ using document root <none> and password file <none>
I20260925 02:59:55.587491 1039 fs_manager.cc:359] Metadata directory not provided
I20260925 02:59:55.587633 1039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:59:55.587919 1039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:59:55.590544 1039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "367161aa0d6441f186711e0eaef22188"
format_stamp: "Formatted at 2026-09-25 02:59:55 on dist-test-slave-8zvk"
server_key: "90ba3374d54814a7246ca94e4c7b10ff"
server_key_iv: "161c30ac7d9e0a670a476c38dc282cc8"
server_key_version: "encryptionkey@0"
I20260925 02:59:55.591301 1039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "367161aa0d6441f186711e0eaef22188"
format_stamp: "Formatted at 2026-09-25 02:59:55 on dist-test-slave-8zvk"
server_key: "90ba3374d54814a7246ca94e4c7b10ff"
server_key_iv: "161c30ac7d9e0a670a476c38dc282cc8"
server_key_version: "encryptionkey@0"
I20260925 02:59:55.596271 1039 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:59:55.599763 1054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:55.600970 1039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:59:55.601135 1039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "367161aa0d6441f186711e0eaef22188"
format_stamp: "Formatted at 2026-09-25 02:59:55 on dist-test-slave-8zvk"
server_key: "90ba3374d54814a7246ca94e4c7b10ff"
server_key_iv: "161c30ac7d9e0a670a476c38dc282cc8"
server_key_version: "encryptionkey@0"
I20260925 02:59:55.601337 1039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:59:55.623926 1039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:59:55.628194 1039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:59:55.628485 1039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:59:55.629463 1039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:59:55.630777 1039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:59:55.630901 1039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 02:59:55.631047 1039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:59:55.631119 1039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:55.663210 1039 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:34925
I20260925 02:59:55.663307 1166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:34925 every 8 connection(s)
I20260925 02:59:55.664561 1039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 02:59:55.674885 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 1039
I20260925 02:59:55.675098 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 02:59:55.675594 19219 external_mini_cluster.cc:1428] Setting key ba90195eff623e8d0e46836466513ad5
Sep 25 02:59:55 dist-test-slave-8zvk krb5kdc[664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305195, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:59:55.683784 901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:55.667105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:44463 (local address 127.18.196.254:39615)
0925 02:59:55.667287 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:55.667299 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:55.668315 (+ 1016us) server_negotiation.cc:407] Connection header received
0925 02:59:55.670087 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:55.670094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:55.670216 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:55.670350 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:55.671801 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:55.672593 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:55.674098 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:55.674579 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:55.674720 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:55.678725 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:55.678730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:55.678733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:55.678764 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:55.681128 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:55.681839 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:55.681843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:55.681846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:55.681963 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:55.682323 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:55.682326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:55.682328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:55.682593 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:55.682701 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:55.683409 (+ 708us) server_negotiation.cc:299] Negotiation successful
0925 02:59:55.683562 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:59:55.684633 1169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:55.667476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.195:44463)
0925 02:59:55.668159 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:55.668194 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:55.669828 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:55.670556 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:55.670564 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:55.671106 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:59:55.671635 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:55.671652 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:55.672754 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:55.672758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:55.673908 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:55.673920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:55.674855 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:55.674863 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:55.675168 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:55.675923 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:59:55.675965 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:59:55.678551 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:59:55.681266 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:55.681276 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:55.681291 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:55.681731 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:55.682053 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:55.682056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:55.682058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:55.682219 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:55.682702 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:59:55.682708 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:59:55.683085 (+ 377us) client_negotiation.cc:769] Sending connection context
0925 02:59:55.683299 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 02:59:55.683639 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":275,"threads_started":1}
I20260925 02:59:55.685928 1167 heartbeater.cc:347] Connected to a master server at 127.18.196.254:39615
I20260925 02:59:55.686177 1167 heartbeater.cc:464] Registering TS with master...
I20260925 02:59:55.686806 1167 heartbeater.cc:511] Master 127.18.196.254:39615 requested a full tablet report, sending...
I20260925 02:59:55.688177 709 ts_manager.cc:194] Registered new tserver with Master: 367161aa0d6441f186711e0eaef22188 (127.18.196.195:34925)
I20260925 02:59:55.688927 709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.195@KRBTEST.COM'} at 127.18.196.195:44463
I20260925 02:59:55.690824 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:59:55.705778 899 heartbeater.cc:503] Master 127.18.196.254:39615 was elected leader, sending a full tablet report...
I20260925 02:59:55.707829 901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:55.695269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41506 (local address 127.18.196.254:39615)
0925 02:59:55.695467 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:55.695476 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:55.695851 (+ 375us) server_negotiation.cc:407] Connection header received
0925 02:59:55.695977 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:55.695980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:55.696039 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:55.696115 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:55.697404 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:55.698146 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:55.699667 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:55.700168 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:55.700263 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:55.702566 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:55.702571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:55.702575 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:55.702603 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:55.704957 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:55.705695 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:55.705699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:55.705702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:55.705880 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:55.706306 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:55.706313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:55.706316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:55.706743 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:55.707020 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:55.707305 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 02:59:55.707538 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:59:55.711604 708 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:59:55.718536 708 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:59:55.742565 1181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:55.730545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49026 (local address 127.18.196.193:32845)
0925 02:59:55.731797 (+ 1252us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:55.731810 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:55.733547 (+ 1737us) server_negotiation.cc:407] Connection header received
0925 02:59:55.733661 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:55.733670 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:55.733919 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:55.734892 (+ 973us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:59:55.735187 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:55.737252 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:55.739775 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:55.741284 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:55.741494 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:55.741669 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:55.742065 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 02:59:55.742230 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1006,"thread_start_us":812,"threads_started":1}
I20260925 02:59:55.742682 1180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:55.730250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:32845 (local address 127.0.0.1:49026)
0925 02:59:55.733029 (+ 2779us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:55.733079 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:55.733309 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:55.734155 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:55.734162 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:55.734206 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:59:55.734647 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:55.734661 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:55.737423 (+ 2762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:55.737430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:55.739621 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:55.739632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:55.741503 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:55.741509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:55.741834 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:55.741879 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 02:59:55.742022 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 02:59:55.742204 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2462,"mutex_wait_us":76,"thread_start_us":3039,"threads_started":1}
I20260925 02:59:55.743693 1182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:55.731044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43120 (local address 127.18.196.194:39683)
0925 02:59:55.734357 (+ 3313us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:55.734368 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:55.734397 (+ 29us) server_negotiation.cc:407] Connection header received
0925 02:59:55.734494 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:55.734500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:55.734709 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:55.735650 (+ 941us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:59:55.735887 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:55.736895 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:55.739944 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:55.741303 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:55.741463 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:55.741669 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:55.743182 (+ 1513us) server_negotiation.cc:299] Negotiation successful
0925 02:59:55.743364 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3124,"thread_start_us":2996,"threads_started":1}
I20260925 02:59:55.744402 1179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:55.730456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:39683 (local address 127.0.0.1:43120)
0925 02:59:55.733626 (+ 3170us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:55.733650 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:55.733788 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:55.734934 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:55.734939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:55.734963 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:59:55.735432 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:55.735446 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:55.737027 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:55.737035 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:55.739777 (+ 2742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:55.739801 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:55.741754 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:55.741760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:55.741970 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:55.742006 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 02:59:55.742100 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 02:59:55.743974 (+ 1874us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3019,"mutex_wait_us":83,"thread_start_us":2023,"threads_started":1}
I20260925 02:59:55.745158 1184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:55.734754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:34925 (local address 127.0.0.1:60360)
0925 02:59:55.737561 (+ 2807us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:55.737582 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:55.737713 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:55.738571 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:55.738577 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:55.738606 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:59:55.739062 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:55.739075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:55.740642 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:55.740647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:55.742470 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:55.742483 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:55.744406 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:55.744413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:55.744580 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:55.744613 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 02:59:55.744695 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:59:55.744830 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2660,"mutex_wait_us":21,"thread_start_us":2441,"threads_started":1}
I20260925 02:59:55.745769 1183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:55.734021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60360 (local address 127.18.196.195:34925)
0925 02:59:55.735874 (+ 1853us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:55.735882 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:55.737692 (+ 1810us) server_negotiation.cc:407] Connection header received
0925 02:59:55.737875 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:55.737882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:55.738097 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:55.738288 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:59:55.739255 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:55.740492 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:55.742655 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:55.744172 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:55.744372 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:55.744551 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:55.745279 (+ 728us) server_negotiation.cc:299] Negotiation successful
0925 02:59:55.745449 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1679,"thread_start_us":406,"threads_started":1}
I20260925 02:59:55.746544 834 tablet_service.cc:1467] Processing CreateTablet for tablet 24717aad8e3f45e9aaea69e99337968a (DEFAULT_TABLE table=test-table [id=73742288d5fc48ccbf90999be293e87d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:59:55.747884 834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24717aad8e3f45e9aaea69e99337968a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:59:55.747450 969 tablet_service.cc:1467] Processing CreateTablet for tablet 24717aad8e3f45e9aaea69e99337968a (DEFAULT_TABLE table=test-table [id=73742288d5fc48ccbf90999be293e87d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:59:55.748703 969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24717aad8e3f45e9aaea69e99337968a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:59:55.749116 1102 tablet_service.cc:1467] Processing CreateTablet for tablet 24717aad8e3f45e9aaea69e99337968a (DEFAULT_TABLE table=test-table [id=73742288d5fc48ccbf90999be293e87d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:59:55.750309 1102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24717aad8e3f45e9aaea69e99337968a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:59:55.759081 1185 tablet_bootstrap.cc:492] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1: Bootstrap starting.
I20260925 02:59:55.762146 1185 tablet_bootstrap.cc:654] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1: Neither blocks nor log segments found. Creating new log.
I20260925 02:59:55.763417 1185 log.cc:824] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1: Log is configured to *not* fsync() on all Append() calls
I20260925 02:59:55.763622 1186 tablet_bootstrap.cc:492] T 24717aad8e3f45e9aaea69e99337968a P 621383aa8000491bbdc36bf8de3e80ba: Bootstrap starting.
I20260925 02:59:55.767917 1187 tablet_bootstrap.cc:492] T 24717aad8e3f45e9aaea69e99337968a P 367161aa0d6441f186711e0eaef22188: Bootstrap starting.
I20260925 02:59:55.768151 1186 tablet_bootstrap.cc:654] T 24717aad8e3f45e9aaea69e99337968a P 621383aa8000491bbdc36bf8de3e80ba: Neither blocks nor log segments found. Creating new log.
I20260925 02:59:55.769423 1185 tablet_bootstrap.cc:492] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1: No bootstrap required, opened a new log
I20260925 02:59:55.769711 1185 ts_tablet_manager.cc:1397] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260925 02:59:55.769742 1186 log.cc:824] T 24717aad8e3f45e9aaea69e99337968a P 621383aa8000491bbdc36bf8de3e80ba: Log is configured to *not* fsync() on all Append() calls
I20260925 02:59:55.771838 1187 tablet_bootstrap.cc:654] T 24717aad8e3f45e9aaea69e99337968a P 367161aa0d6441f186711e0eaef22188: Neither blocks nor log segments found. Creating new log.
I20260925 02:59:55.773201 1187 log.cc:824] T 24717aad8e3f45e9aaea69e99337968a P 367161aa0d6441f186711e0eaef22188: Log is configured to *not* fsync() on all Append() calls
I20260925 02:59:55.774597 1186 tablet_bootstrap.cc:492] T 24717aad8e3f45e9aaea69e99337968a P 621383aa8000491bbdc36bf8de3e80ba: No bootstrap required, opened a new log
I20260925 02:59:55.774955 1186 ts_tablet_manager.cc:1397] T 24717aad8e3f45e9aaea69e99337968a P 621383aa8000491bbdc36bf8de3e80ba: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260925 02:59:55.777026 1187 tablet_bootstrap.cc:492] T 24717aad8e3f45e9aaea69e99337968a P 367161aa0d6441f186711e0eaef22188: No bootstrap required, opened a new log
I20260925 02:59:55.777302 1187 ts_tablet_manager.cc:1397] T 24717aad8e3f45e9aaea69e99337968a P 367161aa0d6441f186711e0eaef22188: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260925 02:59:55.779682 1185 raft_consensus.cc:348] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83773d3745a43149e1afdf48c90e0b1" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 32845 } } peers { permanent_uuid: "621383aa8000491bbdc36bf8de3e80ba" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39683 } } peers { permanent_uuid: "367161aa0d6441f186711e0eaef22188" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 34925 } }
I20260925 02:59:55.780125 1185 raft_consensus.cc:374] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:59:55.780229 1185 raft_consensus.cc:729] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e83773d3745a43149e1afdf48c90e0b1, State: Initialized, Role: FOLLOWER
I20260925 02:59:55.780822 1185 consensus_queue.cc:260] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83773d3745a43149e1afdf48c90e0b1" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 32845 } } peers { permanent_uuid: "621383aa8000491bbdc36bf8de3e80ba" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39683 } } peers { permanent_uuid: "367161aa0d6441f186711e0eaef22188" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 34925 } }
I20260925 02:59:55.782567 1185 ts_tablet_manager.cc:1428] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260925 02:59:55.785192 1186 raft_consensus.cc:348] T 24717aad8e3f45e9aaea69e99337968a P 621383aa8000491bbdc36bf8de3e80ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83773d3745a43149e1afdf48c90e0b1" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 32845 } } peers { permanent_uuid: "621383aa8000491bbdc36bf8de3e80ba" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39683 } } peers { permanent_uuid: "367161aa0d6441f186711e0eaef22188" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 34925 } }
I20260925 02:59:55.785698 1186 raft_consensus.cc:374] T 24717aad8e3f45e9aaea69e99337968a P 621383aa8000491bbdc36bf8de3e80ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:59:55.785538 1187 raft_consensus.cc:348] T 24717aad8e3f45e9aaea69e99337968a P 367161aa0d6441f186711e0eaef22188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83773d3745a43149e1afdf48c90e0b1" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 32845 } } peers { permanent_uuid: "621383aa8000491bbdc36bf8de3e80ba" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39683 } } peers { permanent_uuid: "367161aa0d6441f186711e0eaef22188" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 34925 } }
I20260925 02:59:55.785825 1186 raft_consensus.cc:729] T 24717aad8e3f45e9aaea69e99337968a P 621383aa8000491bbdc36bf8de3e80ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 621383aa8000491bbdc36bf8de3e80ba, State: Initialized, Role: FOLLOWER
I20260925 02:59:55.785969 1187 raft_consensus.cc:374] T 24717aad8e3f45e9aaea69e99337968a P 367161aa0d6441f186711e0eaef22188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:59:55.786077 1187 raft_consensus.cc:729] T 24717aad8e3f45e9aaea69e99337968a P 367161aa0d6441f186711e0eaef22188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 367161aa0d6441f186711e0eaef22188, State: Initialized, Role: FOLLOWER
I20260925 02:59:55.786552 1186 consensus_queue.cc:260] T 24717aad8e3f45e9aaea69e99337968a P 621383aa8000491bbdc36bf8de3e80ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83773d3745a43149e1afdf48c90e0b1" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 32845 } } peers { permanent_uuid: "621383aa8000491bbdc36bf8de3e80ba" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39683 } } peers { permanent_uuid: "367161aa0d6441f186711e0eaef22188" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 34925 } }
I20260925 02:59:55.786650 1187 consensus_queue.cc:260] T 24717aad8e3f45e9aaea69e99337968a P 367161aa0d6441f186711e0eaef22188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83773d3745a43149e1afdf48c90e0b1" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 32845 } } peers { permanent_uuid: "621383aa8000491bbdc36bf8de3e80ba" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39683 } } peers { permanent_uuid: "367161aa0d6441f186711e0eaef22188" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 34925 } }
I20260925 02:59:55.788018 1186 ts_tablet_manager.cc:1428] T 24717aad8e3f45e9aaea69e99337968a P 621383aa8000491bbdc36bf8de3e80ba: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260925 02:59:55.788450 1034 heartbeater.cc:503] Master 127.18.196.254:39615 was elected leader, sending a full tablet report...
I20260925 02:59:55.789183 1167 heartbeater.cc:503] Master 127.18.196.254:39615 was elected leader, sending a full tablet report...
I20260925 02:59:55.789585 1187 ts_tablet_manager.cc:1428] T 24717aad8e3f45e9aaea69e99337968a P 367161aa0d6441f186711e0eaef22188: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260925 02:59:55.916085 1168 tablet.cc:2369] T 24717aad8e3f45e9aaea69e99337968a P 367161aa0d6441f186711e0eaef22188: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:59:55.920910 1035 tablet.cc:2369] T 24717aad8e3f45e9aaea69e99337968a P 621383aa8000491bbdc36bf8de3e80ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:59:55.926484 900 tablet.cc:2369] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:59:56.050006 1191 raft_consensus.cc:482] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:59:56.050345 1191 raft_consensus.cc:504] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83773d3745a43149e1afdf48c90e0b1" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 32845 } } peers { permanent_uuid: "621383aa8000491bbdc36bf8de3e80ba" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39683 } } peers { permanent_uuid: "367161aa0d6441f186711e0eaef22188" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 34925 } }
I20260925 02:59:56.052042 1191 leader_election.cc:288] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 621383aa8000491bbdc36bf8de3e80ba (127.18.196.194:39683), 367161aa0d6441f186711e0eaef22188 (127.18.196.195:34925)
I20260925 02:59:56.061496 1182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:56.052595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:35295 (local address 127.18.196.194:39683)
0925 02:59:56.052812 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:56.052821 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:56.053414 (+ 593us) server_negotiation.cc:407] Connection header received
0925 02:59:56.053619 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:56.053625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:56.053703 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:56.053799 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:59:56.054682 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:56.055864 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:56.058989 (+ 3125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:56.060327 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:56.060493 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:56.060567 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:56.061114 (+ 547us) server_negotiation.cc:299] Negotiation successful
0925 02:59:56.061265 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:59:56.061556 1194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:56.052431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:34925 (local address 127.18.196.193:37927)
0925 02:59:56.053284 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:56.053304 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:56.053468 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:56.053833 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:56.053838 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:56.053870 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:59:56.054525 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:56.054536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:56.056013 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:56.056019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:56.058738 (+ 2719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:56.058754 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:56.060590 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:56.060598 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:56.060859 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:56.060901 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 02:59:56.060969 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 02:59:56.061106 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":263,"spinlock_wait_cycles":15616,"thread_start_us":482,"threads_started":1}
I20260925 02:59:56.061683 1195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:56.052537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:39683 (local address 127.18.196.193:35295)
0925 02:59:56.053284 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:56.053304 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:56.053468 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:56.053833 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:56.053838 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:56.053872 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:59:56.054523 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:56.054536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:56.056014 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:56.056019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:56.058738 (+ 2719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:56.058754 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:56.060606 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:56.060611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:56.060930 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:56.060965 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 02:59:56.061067 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 02:59:56.061246 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":339,"spinlock_wait_cycles":26496,"thread_start_us":384,"threads_started":1}
I20260925 02:59:56.062580 1183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:56.052475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:37927 (local address 127.18.196.195:34925)
0925 02:59:56.052684 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:56.052689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:56.053418 (+ 729us) server_negotiation.cc:407] Connection header received
0925 02:59:56.053619 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:56.053625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:56.053703 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:56.053808 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:59:56.054682 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:56.055864 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:56.058995 (+ 3131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:56.060328 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:56.062074 (+ 1746us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:56.062171 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:56.062223 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 02:59:56.062347 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 02:59:56.062769 989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24717aad8e3f45e9aaea69e99337968a" candidate_uuid: "e83773d3745a43149e1afdf48c90e0b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "621383aa8000491bbdc36bf8de3e80ba" is_pre_election: true
I20260925 02:59:56.063378 989 raft_consensus.cc:2454] T 24717aad8e3f45e9aaea69e99337968a P 621383aa8000491bbdc36bf8de3e80ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e83773d3745a43149e1afdf48c90e0b1 in term 0.
I20260925 02:59:56.063546 1122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24717aad8e3f45e9aaea69e99337968a" candidate_uuid: "e83773d3745a43149e1afdf48c90e0b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "367161aa0d6441f186711e0eaef22188" is_pre_election: true
I20260925 02:59:56.063983 787 leader_election.cc:302] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 621383aa8000491bbdc36bf8de3e80ba, e83773d3745a43149e1afdf48c90e0b1; no voters:
I20260925 02:59:56.063975 1122 raft_consensus.cc:2454] T 24717aad8e3f45e9aaea69e99337968a P 367161aa0d6441f186711e0eaef22188 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e83773d3745a43149e1afdf48c90e0b1 in term 0.
I20260925 02:59:56.064337 1191 raft_consensus.cc:2781] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:59:56.064538 1191 raft_consensus.cc:482] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:59:56.064644 1191 raft_consensus.cc:3037] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:59:56.066828 1191 raft_consensus.cc:504] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83773d3745a43149e1afdf48c90e0b1" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 32845 } } peers { permanent_uuid: "621383aa8000491bbdc36bf8de3e80ba" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39683 } } peers { permanent_uuid: "367161aa0d6441f186711e0eaef22188" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 34925 } }
I20260925 02:59:56.067548 1191 leader_election.cc:288] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1 [CANDIDATE]: Term 1 election: Requested vote from peers 621383aa8000491bbdc36bf8de3e80ba (127.18.196.194:39683), 367161aa0d6441f186711e0eaef22188 (127.18.196.195:34925)
I20260925 02:59:56.067986 989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24717aad8e3f45e9aaea69e99337968a" candidate_uuid: "e83773d3745a43149e1afdf48c90e0b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "621383aa8000491bbdc36bf8de3e80ba"
I20260925 02:59:56.068070 1122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24717aad8e3f45e9aaea69e99337968a" candidate_uuid: "e83773d3745a43149e1afdf48c90e0b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "367161aa0d6441f186711e0eaef22188"
I20260925 02:59:56.068212 989 raft_consensus.cc:3037] T 24717aad8e3f45e9aaea69e99337968a P 621383aa8000491bbdc36bf8de3e80ba [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:59:56.068305 1122 raft_consensus.cc:3037] T 24717aad8e3f45e9aaea69e99337968a P 367161aa0d6441f186711e0eaef22188 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:59:56.071484 989 raft_consensus.cc:2454] T 24717aad8e3f45e9aaea69e99337968a P 621383aa8000491bbdc36bf8de3e80ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e83773d3745a43149e1afdf48c90e0b1 in term 1.
I20260925 02:59:56.071484 1122 raft_consensus.cc:2454] T 24717aad8e3f45e9aaea69e99337968a P 367161aa0d6441f186711e0eaef22188 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e83773d3745a43149e1afdf48c90e0b1 in term 1.
I20260925 02:59:56.071887 787 leader_election.cc:302] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 621383aa8000491bbdc36bf8de3e80ba, e83773d3745a43149e1afdf48c90e0b1; no voters:
I20260925 02:59:56.072213 1191 raft_consensus.cc:2781] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:59:56.072705 1191 raft_consensus.cc:686] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1 [term 1 LEADER]: Becoming Leader. State: Replica: e83773d3745a43149e1afdf48c90e0b1, State: Running, Role: LEADER
I20260925 02:59:56.073271 1191 consensus_queue.cc:237] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83773d3745a43149e1afdf48c90e0b1" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 32845 } } peers { permanent_uuid: "621383aa8000491bbdc36bf8de3e80ba" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39683 } } peers { permanent_uuid: "367161aa0d6441f186711e0eaef22188" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 34925 } }
I20260925 02:59:56.078052 709 catalog_manager.cc:5495] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to e83773d3745a43149e1afdf48c90e0b1 (127.18.196.193). New cstate: current_term: 1 leader_uuid: "e83773d3745a43149e1afdf48c90e0b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83773d3745a43149e1afdf48c90e0b1" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 32845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "621383aa8000491bbdc36bf8de3e80ba" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "367161aa0d6441f186711e0eaef22188" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 34925 } health_report { overall_health: UNKNOWN } } }
I20260925 02:59:56.111967 1181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:56.106386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49040 (local address 127.18.196.193:32845)
0925 02:59:56.106574 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:56.106579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:56.106594 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:59:56.106679 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:56.106683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:56.106763 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:56.106979 (+ 216us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:59:56.107387 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:56.108198 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:56.109638 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:56.110051 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:56.110186 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:56.110682 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:59:56.111265 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:59:56.111426 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:56.111517 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 02:59:56.111620 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:59:56.119937 1122 raft_consensus.cc:1261] T 24717aad8e3f45e9aaea69e99337968a P 367161aa0d6441f186711e0eaef22188 [term 1 FOLLOWER]: Refusing update from remote peer e83773d3745a43149e1afdf48c90e0b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:59:56.119937 989 raft_consensus.cc:1261] T 24717aad8e3f45e9aaea69e99337968a P 621383aa8000491bbdc36bf8de3e80ba [term 1 FOLLOWER]: Refusing update from remote peer e83773d3745a43149e1afdf48c90e0b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:59:56.120867 1196 consensus_queue.cc:1035] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "621383aa8000491bbdc36bf8de3e80ba" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:59:56.121207 1191 consensus_queue.cc:1035] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "367161aa0d6441f186711e0eaef22188" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 34925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:59:56.132747 1199 mvcc.cc:205] Tried to move back new op lower bound from 7333090083297865728 to 7333090083119943680. Current Snapshot: MvccSnapshot[applied={T|T < 7333090083297865728}]
I20260925 02:59:56.147508 709 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:59:56.147807 709 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:59:56.151134 709 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96: Sending DeleteTablet for 3 replicas of tablet 24717aad8e3f45e9aaea69e99337968a
I20260925 02:59:56.152256 834 tablet_service.cc:1514] Processing DeleteTablet for tablet 24717aad8e3f45e9aaea69e99337968a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:59:56 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:49026
I20260925 02:59:56.152361 969 tablet_service.cc:1514] Processing DeleteTablet for tablet 24717aad8e3f45e9aaea69e99337968a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:59:56 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:43120
I20260925 02:59:56.152930 1102 tablet_service.cc:1514] Processing DeleteTablet for tablet 24717aad8e3f45e9aaea69e99337968a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:59:56 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:60360
I20260925 02:59:56.153172 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 770
I20260925 02:59:56.168316 1210 tablet_replica.cc:330] T 24717aad8e3f45e9aaea69e99337968a P 621383aa8000491bbdc36bf8de3e80ba: stopping tablet replica
I20260925 02:59:56.168911 1210 raft_consensus.cc:2229] T 24717aad8e3f45e9aaea69e99337968a P 621383aa8000491bbdc36bf8de3e80ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:59:56.169569 1210 raft_consensus.cc:2258] T 24717aad8e3f45e9aaea69e99337968a P 621383aa8000491bbdc36bf8de3e80ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:59:56.170153 1211 tablet_replica.cc:330] T 24717aad8e3f45e9aaea69e99337968a P 367161aa0d6441f186711e0eaef22188: stopping tablet replica
I20260925 02:59:56.170899 1211 raft_consensus.cc:2229] T 24717aad8e3f45e9aaea69e99337968a P 367161aa0d6441f186711e0eaef22188 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:59:56.171427 1211 raft_consensus.cc:2258] T 24717aad8e3f45e9aaea69e99337968a P 367161aa0d6441f186711e0eaef22188 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:59:56.172242 1210 ts_tablet_manager.cc:1905] T 24717aad8e3f45e9aaea69e99337968a P 621383aa8000491bbdc36bf8de3e80ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:59:56.173941 1211 ts_tablet_manager.cc:1905] T 24717aad8e3f45e9aaea69e99337968a P 367161aa0d6441f186711e0eaef22188: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:59:56.179313 1210 ts_tablet_manager.cc:1918] T 24717aad8e3f45e9aaea69e99337968a P 621383aa8000491bbdc36bf8de3e80ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:59:56.179543 1210 log.cc:1196] T 24717aad8e3f45e9aaea69e99337968a P 621383aa8000491bbdc36bf8de3e80ba: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/24717aad8e3f45e9aaea69e99337968a
I20260925 02:59:56.180099 1210 ts_tablet_manager.cc:1939] T 24717aad8e3f45e9aaea69e99337968a P 621383aa8000491bbdc36bf8de3e80ba: Deleting consensus metadata
I20260925 02:59:56.181809 1209 tablet_replica.cc:330] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1: stopping tablet replica
I20260925 02:59:56.181931 694 catalog_manager.cc:4841] TS 621383aa8000491bbdc36bf8de3e80ba (127.18.196.194:39683): tablet 24717aad8e3f45e9aaea69e99337968a (table test-table [id=73742288d5fc48ccbf90999be293e87d]) successfully deleted
I20260925 02:59:56.182336 1209 raft_consensus.cc:2229] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:59:56.183036 1209 raft_consensus.cc:2258] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:59:56.183472 1211 ts_tablet_manager.cc:1918] T 24717aad8e3f45e9aaea69e99337968a P 367161aa0d6441f186711e0eaef22188: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:59:56.183683 1211 log.cc:1196] T 24717aad8e3f45e9aaea69e99337968a P 367161aa0d6441f186711e0eaef22188: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/24717aad8e3f45e9aaea69e99337968a
I20260925 02:59:56.184247 1211 ts_tablet_manager.cc:1939] T 24717aad8e3f45e9aaea69e99337968a P 367161aa0d6441f186711e0eaef22188: Deleting consensus metadata
I20260925 02:59:56.185181 1209 ts_tablet_manager.cc:1905] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:59:56.186149 693 catalog_manager.cc:4841] TS 367161aa0d6441f186711e0eaef22188 (127.18.196.195:34925): tablet 24717aad8e3f45e9aaea69e99337968a (table test-table [id=73742288d5fc48ccbf90999be293e87d]) successfully deleted
I20260925 02:59:56.192937 1209 ts_tablet_manager.cc:1918] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:59:56.193186 1209 log.cc:1196] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/wal/wals/24717aad8e3f45e9aaea69e99337968a
I20260925 02:59:56.193877 1209 ts_tablet_manager.cc:1939] T 24717aad8e3f45e9aaea69e99337968a P e83773d3745a43149e1afdf48c90e0b1: Deleting consensus metadata
I20260925 02:59:56.196184 693 catalog_manager.cc:4841] TS e83773d3745a43149e1afdf48c90e0b1 (127.18.196.193:32845): tablet 24717aad8e3f45e9aaea69e99337968a (table test-table [id=73742288d5fc48ccbf90999be293e87d]) successfully deleted
Sep 25 02:59:56 dist-test-slave-8zvk krb5kdc[664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.193@KRBTEST.COM
I20260925 02:59:56.207567 1181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:56.154032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49056 (local address 127.18.196.193:32845)
0925 02:59:56.154258 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:56.154267 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:56.161756 (+ 7489us) server_negotiation.cc:407] Connection header received
0925 02:59:56.162942 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:56.162951 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:56.163082 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:56.163174 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:56.168599 (+ 5425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:56.169756 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:56.193315 (+ 23559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:56.194117 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:56.194240 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:56.199905 (+ 5665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:56.199924 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:56.199947 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:56.199990 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:56.203969 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:56.204828 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:56.204836 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:56.204838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:56.205090 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:56.205670 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:56.205676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:56.205679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:56.206217 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:56.206424 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:56.206959 (+ 535us) server_negotiation.cc:299] Negotiation successful
0925 02:59:56.207200 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:59:56.208284 894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:59:56.309422 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 770
I20260925 02:59:56.328898 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 906
Sep 25 02:59:56 dist-test-slave-8zvk krb5kdc[664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.194@KRBTEST.COM
I20260925 02:59:56.343941 1182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:56.329632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43130 (local address 127.18.196.194:39683)
0925 02:59:56.329835 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:56.329842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:56.329935 (+ 93us) server_negotiation.cc:407] Connection header received
0925 02:59:56.330143 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:56.330147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:56.330221 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:56.330330 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:56.331695 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:56.332394 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:56.333760 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:56.334185 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:56.334317 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:56.338570 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:56.338583 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:56.338599 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:56.338637 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:56.341193 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:56.341879 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:56.341886 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:56.341889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:56.342017 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:56.342399 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:56.342404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:56.342406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:56.342760 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:56.343085 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:56.343388 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 02:59:56.343609 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:59:56.344566 1029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:59:56.434544 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 906
I20260925 02:59:56.451574 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 1039
Sep 25 02:59:56 dist-test-slave-8zvk krb5kdc[664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.195@KRBTEST.COM
I20260925 02:59:56.466325 1183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:56.452406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60366 (local address 127.18.196.195:34925)
0925 02:59:56.452601 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:56.452606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:56.452624 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:59:56.452759 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:56.452762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:56.452826 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:56.452908 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:56.454173 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:56.454927 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:56.456228 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:56.456615 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:56.456711 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:56.461088 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:56.461103 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:56.461119 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:56.461151 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:56.463851 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:56.464456 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:56.464463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:56.464465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:56.464609 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:56.464944 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:56.464947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:56.464949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:56.465309 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:56.465520 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:56.465792 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 02:59:56.465951 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:59:56.467096 1162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:59:56.556192 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 1039
I20260925 02:59:56.573920 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 677
I20260925 02:59:56.575109 739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:59:56.687491 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 677
I20260925 02:59:56.709569 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:39615
--webserver_interface=127.18.196.254
--webserver_port=46265
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:37687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:39615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:59:57.032763 1217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:59:57.033030 1217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:59:57.033129 1217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:59:57.040772 1217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:59:57.040865 1217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:59:57.040910 1217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:59:57.040966 1217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:59:57.041008 1217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:59:57.050411 1217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:37687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:39615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:39615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=46265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:59:57.051622 1217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:59:57.052827 1217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:59:57.075939 1223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:57.076078 1224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:57.076014 1226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:59:57.077203 1217 server_base.cc:1034] running on GCE node
I20260925 02:59:57.078102 1217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:59:57.079744 1217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:59:57.080987 1217 hybrid_clock.cc:648] HybridClock initialized: now 1790305197080929 us; error 62 us; skew 500 ppm
I20260925 02:59:57.081379 1217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:59:57 dist-test-slave-8zvk krb5kdc[664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305197, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:59:57.085325 1217 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 02:59:57.087059 1217 webserver.cc:460] Webserver started at http://127.18.196.254:46265/ using document root <none> and password file <none>
I20260925 02:59:57.087769 1217 fs_manager.cc:359] Metadata directory not provided
I20260925 02:59:57.087886 1217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:59:57.093827 1217 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260925 02:59:57.096603 1233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:57.097926 1217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:59:57.098068 1217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "fb2916c744e440ec97ee79298c2eed96"
format_stamp: "Formatted at 2026-09-25 02:59:54 on dist-test-slave-8zvk"
server_key: "94a7defae33de466c27aae988bc5bbbc"
server_key_iv: "c6c94a417f02982cb0a44c2fab4f0bff"
server_key_version: "encryptionkey@0"
I20260925 02:59:57.099151 1217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:59:57.127061 1217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:59:57.131443 1217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:59:57.131661 1217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:59:57.150617 1217 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:39615
I20260925 02:59:57.150655 1284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:39615 every 8 connection(s)
I20260925 02:59:57.152113 1217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:59:57.158782 1285 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:59:57.161571 1285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96: Bootstrap starting.
I20260925 02:59:57.162209 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 1217
I20260925 02:59:57.163260 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:32845
--local_ip_for_outbound_sockets=127.18.196.193
--tserver_master_addrs=127.18.196.254:39615
--webserver_port=40759
--webserver_interface=127.18.196.193
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:37687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:59:57.183928 1285 log.cc:824] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96: Log is configured to *not* fsync() on all Append() calls
I20260925 02:59:57.195444 1285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:59:57.195948 1285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96: Bootstrap complete.
I20260925 02:59:57.204950 1285 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2916c744e440ec97ee79298c2eed96" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39615 } }
I20260925 02:59:57.205739 1285 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb2916c744e440ec97ee79298c2eed96, State: Initialized, Role: FOLLOWER
I20260925 02:59:57.206327 1285 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2916c744e440ec97ee79298c2eed96" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39615 } }
I20260925 02:59:57.206569 1285 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:59:57.206691 1285 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:59:57.206904 1285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:59:57.210551 1285 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2916c744e440ec97ee79298c2eed96" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39615 } }
I20260925 02:59:57.211120 1285 leader_election.cc:302] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb2916c744e440ec97ee79298c2eed96; no voters:
I20260925 02:59:57.211742 1285 leader_election.cc:288] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:59:57.211869 1289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:59:57.214200 1285 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:59:57.218367 1289 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 2 LEADER]: Becoming Leader. State: Replica: fb2916c744e440ec97ee79298c2eed96, State: Running, Role: LEADER
I20260925 02:59:57.218945 1289 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2916c744e440ec97ee79298c2eed96" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39615 } }
I20260925 02:59:57.221953 1291 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb2916c744e440ec97ee79298c2eed96. Latest consensus state: current_term: 2 leader_uuid: "fb2916c744e440ec97ee79298c2eed96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2916c744e440ec97ee79298c2eed96" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39615 } } }
I20260925 02:59:57.222342 1291 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [sys.catalog]: This master's current role is: LEADER
I20260925 02:59:57.222877 1290 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fb2916c744e440ec97ee79298c2eed96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2916c744e440ec97ee79298c2eed96" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39615 } } }
I20260925 02:59:57.223145 1290 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [sys.catalog]: This master's current role is: LEADER
I20260925 02:59:57.231448 1295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:59:57.240026 1295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:59:57.241560 1295 catalog_manager.cc:1259] Loaded cluster ID: e0ba09d6c9704269960ffab4fa1c5745
I20260925 02:59:57.241763 1295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:59:57.244725 1295 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 02:59:57.262712 1295 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 02:59:57.284087 1295 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:59:57.285842 1295 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96: Loaded TSK: 0
I20260925 02:59:57.287238 1295 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:59:57.506251 1287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:59:57.506515 1287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:59:57.506673 1287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:59:57.513671 1287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:59:57.513799 1287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:59:57.513980 1287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 02:59:57.523236 1287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:37687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:32845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=40759
--tserver_master_addrs=127.18.196.254:39615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:59:57.524260 1287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:59:57.525411 1287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:59:57.539088 1313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:57.539088 1314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:57.539746 1316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:59:57.539976 1287 server_base.cc:1034] running on GCE node
I20260925 02:59:57.540597 1287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:59:57.541600 1287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:59:57.542874 1287 hybrid_clock.cc:648] HybridClock initialized: now 1790305197542775 us; error 44 us; skew 500 ppm
I20260925 02:59:57.543184 1287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:59:57 dist-test-slave-8zvk krb5kdc[664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305197, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:59:57.546324 1287 init.cc:377] Logged in from keytab as kudu/127.18.196.193@KRBTEST.COM (short username kudu)
I20260925 02:59:57.547814 1287 webserver.cc:460] Webserver started at http://127.18.196.193:40759/ using document root <none> and password file <none>
I20260925 02:59:57.548453 1287 fs_manager.cc:359] Metadata directory not provided
I20260925 02:59:57.548623 1287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:59:57.553702 1287 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 02:59:57.556288 1323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:57.557596 1287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 02:59:57.557770 1287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "e83773d3745a43149e1afdf48c90e0b1"
format_stamp: "Formatted at 2026-09-25 02:59:54 on dist-test-slave-8zvk"
server_key: "7a4cf11b744cb2c6c3f4e0028439c259"
server_key_iv: "49b8041bf5ddbd544c14ba973d85c19f"
server_key_version: "encryptionkey@0"
I20260925 02:59:57.558703 1287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:59:57.573645 1287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:59:57.577288 1287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:59:57.577538 1287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:59:57.578472 1287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:59:57.579902 1287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:59:57.580040 1287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:57.580188 1287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:59:57.580277 1287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:57.614904 1287 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:32845
I20260925 02:59:57.614965 1435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:32845 every 8 connection(s)
I20260925 02:59:57.616248 1287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 02:59:57.620052 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 1287
I20260925 02:59:57.620666 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:39683
--local_ip_for_outbound_sockets=127.18.196.194
--tserver_master_addrs=127.18.196.254:39615
--webserver_port=35843
--webserver_interface=127.18.196.194
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:37687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:59:57 dist-test-slave-8zvk krb5kdc[664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305197, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:59:57.640298 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:57.618980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.193:45649)
0925 02:59:57.619749 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:57.619800 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:57.621549 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:57.623095 (+ 1546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:57.623102 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:57.623557 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:59:57.624055 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:57.624071 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:57.626348 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:57.626352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:57.627628 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:57.627642 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:57.628915 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:57.628924 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:57.629258 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:57.630010 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:59:57.630043 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:59:57.632602 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:59:57.636012 (+ 3410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:57.636021 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:57.636034 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:57.636444 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:57.636887 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:57.636893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:57.636896 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:57.637093 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:57.637814 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:59:57.637821 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:59:57.638143 (+ 322us) client_negotiation.cc:769] Sending connection context
0925 02:59:57.638386 (+ 243us) client_negotiation.cc:240] Negotiation successful
0925 02:59:57.638994 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":325,"threads_started":1}
I20260925 02:59:57.640307 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:57.618788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:45649 (local address 127.18.196.254:39615)
0925 02:59:57.619424 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:57.619434 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:57.619943 (+ 509us) server_negotiation.cc:407] Connection header received
0925 02:59:57.622141 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:57.622160 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:57.622560 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:57.622961 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:57.624287 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:57.626196 (+ 1909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:57.627771 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:57.628680 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:57.628810 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:57.632879 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:57.632887 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:57.632900 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:57.632934 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:57.635861 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:57.636606 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:57.636612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:57.636619 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:57.636768 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:57.637211 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:57.637221 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:57.637223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:57.637675 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:57.637884 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:57.638591 (+ 707us) server_negotiation.cc:299] Negotiation successful
0925 02:59:57.639022 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":261,"threads_started":1}
I20260925 02:59:57.643146 1436 heartbeater.cc:347] Connected to a master server at 127.18.196.254:39615
I20260925 02:59:57.643496 1436 heartbeater.cc:464] Registering TS with master...
I20260925 02:59:57.644452 1436 heartbeater.cc:511] Master 127.18.196.254:39615 requested a full tablet report, sending...
I20260925 02:59:57.646709 1250 ts_manager.cc:194] Registered new tserver with Master: e83773d3745a43149e1afdf48c90e0b1 (127.18.196.193:32845)
I20260925 02:59:57.648160 1250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.193@KRBTEST.COM'} at 127.18.196.193:45649
W20260925 02:59:57.943161 1440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:59:57.943398 1440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:59:57.943552 1440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:59:57.950485 1440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:59:57.950582 1440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:59:57.950795 1440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 02:59:57.959713 1440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:37687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:39683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=35843
--tserver_master_addrs=127.18.196.254:39615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:59:57.960863 1440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:59:57.962095 1440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:59:57.974417 1445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:57.974367 1446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:57.974285 1448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:59:57.975448 1440 server_base.cc:1034] running on GCE node
I20260925 02:59:57.975991 1440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:59:57.976835 1440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:59:57.978062 1440 hybrid_clock.cc:648] HybridClock initialized: now 1790305197978012 us; error 42 us; skew 500 ppm
I20260925 02:59:57.978355 1440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:59:57 dist-test-slave-8zvk krb5kdc[664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305197, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:59:57.981208 1440 init.cc:377] Logged in from keytab as kudu/127.18.196.194@KRBTEST.COM (short username kudu)
I20260925 02:59:57.982610 1440 webserver.cc:460] Webserver started at http://127.18.196.194:35843/ using document root <none> and password file <none>
I20260925 02:59:57.983299 1440 fs_manager.cc:359] Metadata directory not provided
I20260925 02:59:57.983446 1440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:59:57.988348 1440 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:59:57.990795 1455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:57.992048 1440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:59:57.992229 1440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "621383aa8000491bbdc36bf8de3e80ba"
format_stamp: "Formatted at 2026-09-25 02:59:55 on dist-test-slave-8zvk"
server_key: "23bd3733b288b8508b3932600d3bcd0c"
server_key_iv: "57e29a9683bad97027e504a57c9a8e06"
server_key_version: "encryptionkey@0"
I20260925 02:59:57.993235 1440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:59:58.022384 1440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:59:58.026230 1440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:59:58.026502 1440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:59:58.027455 1440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:59:58.028738 1440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:59:58.028872 1440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:58.029001 1440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:59:58.029089 1440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:58.061429 1440 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:39683
I20260925 02:59:58.061565 1568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:39683 every 8 connection(s)
I20260925 02:59:58.062837 1440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 02:59:58.065593 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 1440
I20260925 02:59:58.066159 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:34925
--local_ip_for_outbound_sockets=127.18.196.195
--tserver_master_addrs=127.18.196.254:39615
--webserver_port=43035
--webserver_interface=127.18.196.195
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:37687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:59:58 dist-test-slave-8zvk krb5kdc[664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305197, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:59:58.082656 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:58.065316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:49849 (local address 127.18.196.254:39615)
0925 02:59:58.065538 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:58.065545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:58.066709 (+ 1164us) server_negotiation.cc:407] Connection header received
0925 02:59:58.068503 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:58.068512 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:58.068609 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:58.068733 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:58.070100 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.071202 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.072659 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.073369 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.073500 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:58.077108 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:58.077112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:58.077116 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:58.077148 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:58.079679 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:58.080378 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:58.080385 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:58.080388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:58.080512 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:58.080932 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:58.080941 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:58.080944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:58.081231 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:58.081425 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:58.082090 (+ 665us) server_negotiation.cc:299] Negotiation successful
0925 02:59:58.082326 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:59:58.083562 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:58.065781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.194:49849)
0925 02:59:58.066543 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:58.066587 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:58.068237 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:58.068938 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:58.068946 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:58.069399 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:59:58.069921 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:58.069938 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.071341 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.071345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:58.072508 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:58.072520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.073630 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.073639 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:58.073955 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:58.074676 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:59:58.074720 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:59:58.076910 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:59:58.079827 (+ 2917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:58.079836 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:58.079850 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:58.080257 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:58.080625 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:58.080630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:58.080633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:58.080819 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:58.081390 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:59:58.081399 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:59:58.081740 (+ 341us) client_negotiation.cc:769] Sending connection context
0925 02:59:58.081994 (+ 254us) client_negotiation.cc:240] Negotiation successful
0925 02:59:58.082481 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":334,"threads_started":1}
I20260925 02:59:58.085146 1569 heartbeater.cc:347] Connected to a master server at 127.18.196.254:39615
I20260925 02:59:58.085510 1569 heartbeater.cc:464] Registering TS with master...
I20260925 02:59:58.086457 1569 heartbeater.cc:511] Master 127.18.196.254:39615 requested a full tablet report, sending...
I20260925 02:59:58.088145 1250 ts_manager.cc:194] Registered new tserver with Master: 621383aa8000491bbdc36bf8de3e80ba (127.18.196.194:39683)
I20260925 02:59:58.089470 1250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.194@KRBTEST.COM'} at 127.18.196.194:49849
W20260925 02:59:58.392692 1572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:59:58.392961 1572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:59:58.393055 1572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:59:58.400463 1572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:59:58.400568 1572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:59:58.400676 1572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 02:59:58.410521 1572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:37687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:34925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=43035
--tserver_master_addrs=127.18.196.254:39615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:59:58.411780 1572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:59:58.413053 1572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:59:58.426422 1577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:58.426422 1578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:58.426471 1580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:59:58.427217 1572 server_base.cc:1034] running on GCE node
I20260925 02:59:58.427834 1572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:59:58.428759 1572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:59:58.429998 1572 hybrid_clock.cc:648] HybridClock initialized: now 1790305198429943 us; error 50 us; skew 500 ppm
I20260925 02:59:58.430300 1572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:59:58 dist-test-slave-8zvk krb5kdc[664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305198, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:59:58.433368 1572 init.cc:377] Logged in from keytab as kudu/127.18.196.195@KRBTEST.COM (short username kudu)
I20260925 02:59:58.434779 1572 webserver.cc:460] Webserver started at http://127.18.196.195:43035/ using document root <none> and password file <none>
I20260925 02:59:58.435449 1572 fs_manager.cc:359] Metadata directory not provided
I20260925 02:59:58.435608 1572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:59:58.440819 1572 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 02:59:58.443465 1587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:58.444711 1572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:59:58.444880 1572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "367161aa0d6441f186711e0eaef22188"
format_stamp: "Formatted at 2026-09-25 02:59:55 on dist-test-slave-8zvk"
server_key: "90ba3374d54814a7246ca94e4c7b10ff"
server_key_iv: "161c30ac7d9e0a670a476c38dc282cc8"
server_key_version: "encryptionkey@0"
I20260925 02:59:58.445848 1572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:59:58.458287 1572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:59:58.461925 1572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:59:58.462183 1572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:59:58.463178 1572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:59:58.464465 1572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:59:58.464584 1572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:58.464712 1572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:59:58.464799 1572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:58.497700 1572 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:34925
I20260925 02:59:58.497789 1700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:34925 every 8 connection(s)
I20260925 02:59:58.499215 1572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 02:59:58.500504 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 1572
I20260925 02:59:58.516886 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:58.501397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.18.196.254:39615)
0925 02:59:58.501648 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:58.501656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:58.502055 (+ 399us) server_negotiation.cc:407] Connection header received
0925 02:59:58.502265 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:58.502270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:58.502360 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:58.502458 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:58.504311 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.505049 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.507045 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.507554 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.507716 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:58.511063 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:58.511068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:58.511070 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:58.511095 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:58.513944 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:58.514596 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:58.514600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:58.514602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:58.514796 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:58.515255 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:58.515260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:58.515262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:58.515664 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:58.515924 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:58.516319 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 02:59:58.516535 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 02:59:58 dist-test-slave-8zvk krb5kdc[664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305198, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 02:59:58.523976 1704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:58.502651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:33035 (local address 127.18.196.254:39615)
0925 02:59:58.507600 (+ 4949us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:58.507604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:58.507618 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:59:58.508973 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:58.508977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:58.509038 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:58.509115 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:58.511549 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.512238 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.514481 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.515151 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.515262 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:58.518875 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:58.518882 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:58.518886 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:58.518920 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:58.521507 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:58.522146 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:58.522149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:58.522151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:58.522262 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:58.522622 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:58.522625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:58.522627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:58.522908 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:58.523013 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:58.523578 (+ 565us) server_negotiation.cc:299] Negotiation successful
0925 02:59:58.523725 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4866,"thread_start_us":4728,"threads_started":1}
I20260925 02:59:58.524852 1705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:58.503441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.195:33035)
0925 02:59:58.505375 (+ 1934us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:58.505420 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:58.508680 (+ 3260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:58.509382 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:58.509393 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:58.510064 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:59:58.510845 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:58.510911 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.512406 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.512412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:58.514324 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:58.514335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.515391 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.515400 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:58.515680 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:58.516364 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:59:58.516389 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:59:58.518617 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:59:58.521658 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:58.521664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:58.521674 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:58.522050 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:58.522355 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:59:58.522359 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:59:58.522361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:59:58.522527 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:59:58.523012 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:59:58.523018 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:59:58.523294 (+ 276us) client_negotiation.cc:769] Sending connection context
0925 02:59:58.523510 (+ 216us) client_negotiation.cc:240] Negotiation successful
0925 02:59:58.523862 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":377,"threads_started":1}
I20260925 02:59:58.526103 1701 heartbeater.cc:347] Connected to a master server at 127.18.196.254:39615
I20260925 02:59:58.526404 1701 heartbeater.cc:464] Registering TS with master...
I20260925 02:59:58.527137 1701 heartbeater.cc:511] Master 127.18.196.254:39615 requested a full tablet report, sending...
I20260925 02:59:58.528443 1250 ts_manager.cc:194] Registered new tserver with Master: 367161aa0d6441f186711e0eaef22188 (127.18.196.195:34925)
I20260925 02:59:58.529167 1250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.195@KRBTEST.COM'} at 127.18.196.195:33035
I20260925 02:59:58.531559 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:59:58.537868 1704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:58.532459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41518 (local address 127.18.196.254:39615)
0925 02:59:58.532627 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:58.532634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:58.533063 (+ 429us) server_negotiation.cc:407] Connection header received
0925 02:59:58.533217 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:58.533220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:58.533339 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:58.533444 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:59:58.533936 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.534614 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.535959 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.536402 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.536497 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:58.536954 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:59:58.537336 (+ 382us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:59:58.537438 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:58.537527 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 02:59:58.537584 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:59:58.538623 1250 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:59:58.545581 1250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:59:58.577992 1711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:58.556413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:39683 (local address 127.0.0.1:43136)
0925 02:59:58.568978 (+ 12565us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:58.568993 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:58.569090 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:58.570105 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:58.570109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:58.570132 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:59:58.570581 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:58.570595 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.572931 (+ 2336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.572939 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:58.575461 (+ 2522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:58.575479 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.577072 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.577078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:58.577246 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:58.577292 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 02:59:58.577380 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 02:59:58.577550 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12427,"mutex_wait_us":133,"thread_start_us":11566,"threads_started":1}
I20260925 02:59:58.578493 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:58.556082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:32845 (local address 127.0.0.1:49070)
0925 02:59:58.567504 (+ 11422us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:58.567552 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:58.567777 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:58.569854 (+ 2077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:58.569864 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:58.569904 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:59:58.570468 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:58.570489 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.572854 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.572862 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:58.575846 (+ 2984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:58.575859 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.577497 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.577502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:58.577845 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:58.577883 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 02:59:58.577984 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 02:59:58.578104 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11129,"mutex_wait_us":667,"thread_start_us":12634,"threads_started":1}
I20260925 02:59:58.578523 1712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:58.556258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49070 (local address 127.18.196.193:32845)
0925 02:59:58.569068 (+ 12810us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:58.569076 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:58.569101 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:59:58.569215 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:58.569225 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:58.569451 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:58.569810 (+ 359us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:59:58.570797 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.572682 (+ 1885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.576015 (+ 3333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.577266 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.577691 (+ 425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:58.577855 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:58.578041 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 02:59:58.578224 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12660,"thread_start_us":13312,"threads_started":1}
I20260925 02:59:58.578647 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:58.556489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.18.196.194:39683)
0925 02:59:58.569562 (+ 13073us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:58.569570 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:58.569601 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:59:58.569693 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:58.569699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:58.569894 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:58.570113 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:59:58.570930 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.572753 (+ 1823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.575631 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.576830 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.577878 (+ 1048us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:58.578049 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:58.578112 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 02:59:58.578290 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12910,"thread_start_us":13042,"threads_started":1}
I20260925 02:59:58.582325 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:58.570474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:34925 (local address 127.0.0.1:60382)
0925 02:59:58.573906 (+ 3432us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:58.573918 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:58.574043 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:58.574805 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:58.574809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:58.574852 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:59:58.575336 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:58.575349 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.576832 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.576836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:58.579821 (+ 2985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:58.579833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.581488 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.581494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:58.581719 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:58.581752 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 02:59:58.581838 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 02:59:58.581948 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3344,"mutex_wait_us":1047,"thread_start_us":3271,"threads_started":1}
I20260925 02:59:58.582356 1715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:58.569409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60382 (local address 127.18.196.195:34925)
0925 02:59:58.573750 (+ 4341us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:58.573757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:58.574011 (+ 254us) server_negotiation.cc:407] Connection header received
0925 02:59:58.574425 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:58.574431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:58.574646 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:58.574812 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:59:58.575563 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.576688 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.579983 (+ 3295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.581277 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.581475 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:58.581693 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:58.581917 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 02:59:58.582075 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4187,"thread_start_us":4062,"threads_started":1}
I20260925 02:59:58.582123 1371 tablet_service.cc:1467] Processing CreateTablet for tablet 279de86fe45949dd9d41bec861b483ee (DEFAULT_TABLE table=test-table [id=6e1c224c761e4ff594aa850ebd0a3132]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:59:58.583525 1371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 279de86fe45949dd9d41bec861b483ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:59:58.583280 1503 tablet_service.cc:1467] Processing CreateTablet for tablet 279de86fe45949dd9d41bec861b483ee (DEFAULT_TABLE table=test-table [id=6e1c224c761e4ff594aa850ebd0a3132]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:59:58.584740 1503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 279de86fe45949dd9d41bec861b483ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:59:58.585623 1636 tablet_service.cc:1467] Processing CreateTablet for tablet 279de86fe45949dd9d41bec861b483ee (DEFAULT_TABLE table=test-table [id=6e1c224c761e4ff594aa850ebd0a3132]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:59:58.586766 1636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 279de86fe45949dd9d41bec861b483ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:59:58.596240 1717 tablet_bootstrap.cc:492] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba: Bootstrap starting.
I20260925 02:59:58.599277 1717 tablet_bootstrap.cc:654] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba: Neither blocks nor log segments found. Creating new log.
I20260925 02:59:58.599630 1718 tablet_bootstrap.cc:492] T 279de86fe45949dd9d41bec861b483ee P e83773d3745a43149e1afdf48c90e0b1: Bootstrap starting.
I20260925 02:59:58.600611 1717 log.cc:824] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba: Log is configured to *not* fsync() on all Append() calls
I20260925 02:59:58.601089 1719 tablet_bootstrap.cc:492] T 279de86fe45949dd9d41bec861b483ee P 367161aa0d6441f186711e0eaef22188: Bootstrap starting.
I20260925 02:59:58.603564 1718 tablet_bootstrap.cc:654] T 279de86fe45949dd9d41bec861b483ee P e83773d3745a43149e1afdf48c90e0b1: Neither blocks nor log segments found. Creating new log.
I20260925 02:59:58.604279 1719 tablet_bootstrap.cc:654] T 279de86fe45949dd9d41bec861b483ee P 367161aa0d6441f186711e0eaef22188: Neither blocks nor log segments found. Creating new log.
I20260925 02:59:58.604866 1718 log.cc:824] T 279de86fe45949dd9d41bec861b483ee P e83773d3745a43149e1afdf48c90e0b1: Log is configured to *not* fsync() on all Append() calls
I20260925 02:59:58.605192 1717 tablet_bootstrap.cc:492] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba: No bootstrap required, opened a new log
I20260925 02:59:58.605351 1719 log.cc:824] T 279de86fe45949dd9d41bec861b483ee P 367161aa0d6441f186711e0eaef22188: Log is configured to *not* fsync() on all Append() calls
I20260925 02:59:58.605506 1717 ts_tablet_manager.cc:1397] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 02:59:58.607890 1718 tablet_bootstrap.cc:492] T 279de86fe45949dd9d41bec861b483ee P e83773d3745a43149e1afdf48c90e0b1: No bootstrap required, opened a new log
I20260925 02:59:58.608144 1718 ts_tablet_manager.cc:1397] T 279de86fe45949dd9d41bec861b483ee P e83773d3745a43149e1afdf48c90e0b1: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 02:59:58.608306 1719 tablet_bootstrap.cc:492] T 279de86fe45949dd9d41bec861b483ee P 367161aa0d6441f186711e0eaef22188: No bootstrap required, opened a new log
I20260925 02:59:58.608543 1719 ts_tablet_manager.cc:1397] T 279de86fe45949dd9d41bec861b483ee P 367161aa0d6441f186711e0eaef22188: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260925 02:59:58.613161 1717 raft_consensus.cc:348] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83773d3745a43149e1afdf48c90e0b1" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 32845 } } peers { permanent_uuid: "621383aa8000491bbdc36bf8de3e80ba" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39683 } } peers { permanent_uuid: "367161aa0d6441f186711e0eaef22188" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 34925 } }
I20260925 02:59:58.613498 1717 raft_consensus.cc:374] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:59:58.613618 1717 raft_consensus.cc:729] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 621383aa8000491bbdc36bf8de3e80ba, State: Initialized, Role: FOLLOWER
I20260925 02:59:58.615249 1717 consensus_queue.cc:260] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83773d3745a43149e1afdf48c90e0b1" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 32845 } } peers { permanent_uuid: "621383aa8000491bbdc36bf8de3e80ba" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39683 } } peers { permanent_uuid: "367161aa0d6441f186711e0eaef22188" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 34925 } }
I20260925 02:59:58.617046 1717 ts_tablet_manager.cc:1428] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260925 02:59:58.617529 1569 heartbeater.cc:503] Master 127.18.196.254:39615 was elected leader, sending a full tablet report...
I20260925 02:59:58.618219 1719 raft_consensus.cc:348] T 279de86fe45949dd9d41bec861b483ee P 367161aa0d6441f186711e0eaef22188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83773d3745a43149e1afdf48c90e0b1" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 32845 } } peers { permanent_uuid: "621383aa8000491bbdc36bf8de3e80ba" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39683 } } peers { permanent_uuid: "367161aa0d6441f186711e0eaef22188" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 34925 } }
I20260925 02:59:58.618604 1719 raft_consensus.cc:374] T 279de86fe45949dd9d41bec861b483ee P 367161aa0d6441f186711e0eaef22188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:59:58.618767 1719 raft_consensus.cc:729] T 279de86fe45949dd9d41bec861b483ee P 367161aa0d6441f186711e0eaef22188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 367161aa0d6441f186711e0eaef22188, State: Initialized, Role: FOLLOWER
I20260925 02:59:58.619472 1719 consensus_queue.cc:260] T 279de86fe45949dd9d41bec861b483ee P 367161aa0d6441f186711e0eaef22188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83773d3745a43149e1afdf48c90e0b1" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 32845 } } peers { permanent_uuid: "621383aa8000491bbdc36bf8de3e80ba" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39683 } } peers { permanent_uuid: "367161aa0d6441f186711e0eaef22188" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 34925 } }
I20260925 02:59:58.621044 1718 raft_consensus.cc:348] T 279de86fe45949dd9d41bec861b483ee P e83773d3745a43149e1afdf48c90e0b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83773d3745a43149e1afdf48c90e0b1" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 32845 } } peers { permanent_uuid: "621383aa8000491bbdc36bf8de3e80ba" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39683 } } peers { permanent_uuid: "367161aa0d6441f186711e0eaef22188" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 34925 } }
I20260925 02:59:58.621582 1718 raft_consensus.cc:374] T 279de86fe45949dd9d41bec861b483ee P e83773d3745a43149e1afdf48c90e0b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:59:58.621791 1718 raft_consensus.cc:729] T 279de86fe45949dd9d41bec861b483ee P e83773d3745a43149e1afdf48c90e0b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e83773d3745a43149e1afdf48c90e0b1, State: Initialized, Role: FOLLOWER
I20260925 02:59:58.622641 1718 consensus_queue.cc:260] T 279de86fe45949dd9d41bec861b483ee P e83773d3745a43149e1afdf48c90e0b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83773d3745a43149e1afdf48c90e0b1" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 32845 } } peers { permanent_uuid: "621383aa8000491bbdc36bf8de3e80ba" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39683 } } peers { permanent_uuid: "367161aa0d6441f186711e0eaef22188" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 34925 } }
I20260925 02:59:58.623526 1701 heartbeater.cc:503] Master 127.18.196.254:39615 was elected leader, sending a full tablet report...
I20260925 02:59:58.625674 1719 ts_tablet_manager.cc:1428] T 279de86fe45949dd9d41bec861b483ee P 367161aa0d6441f186711e0eaef22188: Time spent starting tablet: real 0.017s user 0.014s sys 0.002s
I20260925 02:59:58.629417 1436 heartbeater.cc:503] Master 127.18.196.254:39615 was elected leader, sending a full tablet report...
I20260925 02:59:58.629947 1718 ts_tablet_manager.cc:1428] T 279de86fe45949dd9d41bec861b483ee P e83773d3745a43149e1afdf48c90e0b1: Time spent starting tablet: real 0.022s user 0.008s sys 0.012s
I20260925 02:59:58.641387 1723 raft_consensus.cc:482] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:59:58.641672 1723 raft_consensus.cc:504] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83773d3745a43149e1afdf48c90e0b1" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 32845 } } peers { permanent_uuid: "621383aa8000491bbdc36bf8de3e80ba" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39683 } } peers { permanent_uuid: "367161aa0d6441f186711e0eaef22188" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 34925 } }
I20260925 02:59:58.643115 1723 leader_election.cc:288] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e83773d3745a43149e1afdf48c90e0b1 (127.18.196.193:32845), 367161aa0d6441f186711e0eaef22188 (127.18.196.195:34925)
I20260925 02:59:58.649946 1712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:58.643603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:60271 (local address 127.18.196.193:32845)
0925 02:59:58.643802 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:58.643808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:58.644050 (+ 242us) server_negotiation.cc:407] Connection header received
0925 02:59:58.644284 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:58.644291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:58.644364 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:58.644429 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:59:58.645243 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.646117 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.648058 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.648972 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.649115 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:58.649182 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:58.649568 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 02:59:58.649715 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:59:58.650043 1726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:58.643367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:32845 (local address 127.18.196.194:60271)
0925 02:59:58.643913 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:58.643955 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:58.644105 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:58.644604 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:58.644607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:58.644639 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:59:58.645072 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:58.645086 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.646244 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.646249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:58.647898 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:58.647910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.649182 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.649191 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:58.649359 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:58.649402 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 02:59:58.649526 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 02:59:58.649626 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":5,"thread_start_us":264,"threads_started":1}
I20260925 02:59:58.650123 1715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:58.644221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:40299 (local address 127.18.196.195:34925)
0925 02:59:58.644403 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:58.644411 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:58.644693 (+ 282us) server_negotiation.cc:407] Connection header received
0925 02:59:58.644839 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:58.644843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:58.644913 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:58.645009 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:59:58.645646 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.646443 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.648274 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.649122 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.649271 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:58.649359 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:58.649736 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 02:59:58.649864 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:59:58.650416 1727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:58.643995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:34925 (local address 127.18.196.194:40299)
0925 02:59:58.644589 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0925 02:59:58.644608 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:59:58.644712 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:59:58.645038 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:59:58.645041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:59:58.645062 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:59:58.645478 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:58.645490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.646585 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.646589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:58.648127 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:59:58.648141 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.649362 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.649370 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:59:58.649573 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:58.649605 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 02:59:58.649692 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 02:59:58.649807 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":17,"thread_start_us":303,"threads_started":1}
I20260925 02:59:58.651372 1391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "279de86fe45949dd9d41bec861b483ee" candidate_uuid: "621383aa8000491bbdc36bf8de3e80ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e83773d3745a43149e1afdf48c90e0b1" is_pre_election: true
I20260925 02:59:58.651428 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "279de86fe45949dd9d41bec861b483ee" candidate_uuid: "621383aa8000491bbdc36bf8de3e80ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "367161aa0d6441f186711e0eaef22188" is_pre_election: true
I20260925 02:59:58.651831 1391 raft_consensus.cc:2454] T 279de86fe45949dd9d41bec861b483ee P e83773d3745a43149e1afdf48c90e0b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 621383aa8000491bbdc36bf8de3e80ba in term 0.
I20260925 02:59:58.651877 1656 raft_consensus.cc:2454] T 279de86fe45949dd9d41bec861b483ee P 367161aa0d6441f186711e0eaef22188 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 621383aa8000491bbdc36bf8de3e80ba in term 0.
I20260925 02:59:58.652469 1456 leader_election.cc:302] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 621383aa8000491bbdc36bf8de3e80ba, e83773d3745a43149e1afdf48c90e0b1; no voters:
I20260925 02:59:58.652830 1723 raft_consensus.cc:2781] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:59:58.652994 1723 raft_consensus.cc:482] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:59:58.653101 1723 raft_consensus.cc:3037] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:59:58.655181 1723 raft_consensus.cc:504] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83773d3745a43149e1afdf48c90e0b1" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 32845 } } peers { permanent_uuid: "621383aa8000491bbdc36bf8de3e80ba" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39683 } } peers { permanent_uuid: "367161aa0d6441f186711e0eaef22188" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 34925 } }
I20260925 02:59:58.655735 1723 leader_election.cc:288] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba [CANDIDATE]: Term 1 election: Requested vote from peers e83773d3745a43149e1afdf48c90e0b1 (127.18.196.193:32845), 367161aa0d6441f186711e0eaef22188 (127.18.196.195:34925)
I20260925 02:59:58.656085 1391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "279de86fe45949dd9d41bec861b483ee" candidate_uuid: "621383aa8000491bbdc36bf8de3e80ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e83773d3745a43149e1afdf48c90e0b1"
I20260925 02:59:58.656229 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "279de86fe45949dd9d41bec861b483ee" candidate_uuid: "621383aa8000491bbdc36bf8de3e80ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "367161aa0d6441f186711e0eaef22188"
I20260925 02:59:58.656291 1391 raft_consensus.cc:3037] T 279de86fe45949dd9d41bec861b483ee P e83773d3745a43149e1afdf48c90e0b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:59:58.656428 1656 raft_consensus.cc:3037] T 279de86fe45949dd9d41bec861b483ee P 367161aa0d6441f186711e0eaef22188 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:59:58.658387 1391 raft_consensus.cc:2454] T 279de86fe45949dd9d41bec861b483ee P e83773d3745a43149e1afdf48c90e0b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 621383aa8000491bbdc36bf8de3e80ba in term 1.
I20260925 02:59:58.658556 1656 raft_consensus.cc:2454] T 279de86fe45949dd9d41bec861b483ee P 367161aa0d6441f186711e0eaef22188 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 621383aa8000491bbdc36bf8de3e80ba in term 1.
I20260925 02:59:58.658775 1456 leader_election.cc:302] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 621383aa8000491bbdc36bf8de3e80ba, e83773d3745a43149e1afdf48c90e0b1; no voters:
I20260925 02:59:58.659132 1723 raft_consensus.cc:2781] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:59:58.659550 1723 raft_consensus.cc:686] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba [term 1 LEADER]: Becoming Leader. State: Replica: 621383aa8000491bbdc36bf8de3e80ba, State: Running, Role: LEADER
I20260925 02:59:58.660029 1723 consensus_queue.cc:237] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83773d3745a43149e1afdf48c90e0b1" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 32845 } } peers { permanent_uuid: "621383aa8000491bbdc36bf8de3e80ba" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39683 } } peers { permanent_uuid: "367161aa0d6441f186711e0eaef22188" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 34925 } }
I20260925 02:59:58.664332 1250 catalog_manager.cc:5495] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 621383aa8000491bbdc36bf8de3e80ba (127.18.196.194). New cstate: current_term: 1 leader_uuid: "621383aa8000491bbdc36bf8de3e80ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83773d3745a43149e1afdf48c90e0b1" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 32845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "621383aa8000491bbdc36bf8de3e80ba" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "367161aa0d6441f186711e0eaef22188" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 34925 } health_report { overall_health: UNKNOWN } } }
I20260925 02:59:58.682403 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:58.677113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43140 (local address 127.18.196.194:39683)
0925 02:59:58.677309 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:58.677314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:58.677331 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:59:58.677541 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:58.677544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:58.677622 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:58.677792 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:59:58.678146 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.678886 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.680338 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.680749 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.680883 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:58.681348 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:59:58.681827 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:59:58.681978 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:58.682077 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 02:59:58.682154 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:59:58.690412 1391 raft_consensus.cc:1261] T 279de86fe45949dd9d41bec861b483ee P e83773d3745a43149e1afdf48c90e0b1 [term 1 FOLLOWER]: Refusing update from remote peer 621383aa8000491bbdc36bf8de3e80ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:59:58.690413 1656 raft_consensus.cc:1261] T 279de86fe45949dd9d41bec861b483ee P 367161aa0d6441f186711e0eaef22188 [term 1 FOLLOWER]: Refusing update from remote peer 621383aa8000491bbdc36bf8de3e80ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:59:58.691383 1728 consensus_queue.cc:1035] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "367161aa0d6441f186711e0eaef22188" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 34925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:59:58.691764 1723 consensus_queue.cc:1035] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "e83773d3745a43149e1afdf48c90e0b1" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 32845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:59:58.704187 1731 mvcc.cc:205] Tried to move back new op lower bound from 7333090093825458176 to 7333090093715222528. Current Snapshot: MvccSnapshot[applied={T|T < 7333090093825458176}]
I20260925 02:59:58.707373 1733 mvcc.cc:205] Tried to move back new op lower bound from 7333090093825458176 to 7333090093715222528. Current Snapshot: MvccSnapshot[applied={T|T < 7333090093825458176}]
I20260925 02:59:58.716320 1250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:59:58.716566 1250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:59:58.719652 1250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96: Sending DeleteTablet for 3 replicas of tablet 279de86fe45949dd9d41bec861b483ee
I20260925 02:59:58.720669 1371 tablet_service.cc:1514] Processing DeleteTablet for tablet 279de86fe45949dd9d41bec861b483ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:59:58 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:49070
I20260925 02:59:58.720718 1503 tablet_service.cc:1514] Processing DeleteTablet for tablet 279de86fe45949dd9d41bec861b483ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:59:58 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:43136
I20260925 02:59:58.721012 1636 tablet_service.cc:1514] Processing DeleteTablet for tablet 279de86fe45949dd9d41bec861b483ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:59:58 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:60382
I20260925 02:59:58.722183 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 1287
I20260925 02:59:58.732249 1743 tablet_replica.cc:330] T 279de86fe45949dd9d41bec861b483ee P 367161aa0d6441f186711e0eaef22188: stopping tablet replica
I20260925 02:59:58.732942 1743 raft_consensus.cc:2229] T 279de86fe45949dd9d41bec861b483ee P 367161aa0d6441f186711e0eaef22188 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:59:58.733659 1743 raft_consensus.cc:2258] T 279de86fe45949dd9d41bec861b483ee P 367161aa0d6441f186711e0eaef22188 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:59:58.734503 1741 tablet_replica.cc:330] T 279de86fe45949dd9d41bec861b483ee P e83773d3745a43149e1afdf48c90e0b1: stopping tablet replica
I20260925 02:59:58.734817 1741 raft_consensus.cc:2229] T 279de86fe45949dd9d41bec861b483ee P e83773d3745a43149e1afdf48c90e0b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:59:58.735350 1741 raft_consensus.cc:2258] T 279de86fe45949dd9d41bec861b483ee P e83773d3745a43149e1afdf48c90e0b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:59:58.736961 1743 ts_tablet_manager.cc:1905] T 279de86fe45949dd9d41bec861b483ee P 367161aa0d6441f186711e0eaef22188: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:59:58.737237 1741 ts_tablet_manager.cc:1905] T 279de86fe45949dd9d41bec861b483ee P e83773d3745a43149e1afdf48c90e0b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:59:58.743880 1743 ts_tablet_manager.cc:1918] T 279de86fe45949dd9d41bec861b483ee P 367161aa0d6441f186711e0eaef22188: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:59:58.744073 1743 log.cc:1196] T 279de86fe45949dd9d41bec861b483ee P 367161aa0d6441f186711e0eaef22188: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/279de86fe45949dd9d41bec861b483ee
I20260925 02:59:58.744458 1741 ts_tablet_manager.cc:1918] T 279de86fe45949dd9d41bec861b483ee P e83773d3745a43149e1afdf48c90e0b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:59:58.744673 1743 ts_tablet_manager.cc:1939] T 279de86fe45949dd9d41bec861b483ee P 367161aa0d6441f186711e0eaef22188: Deleting consensus metadata
I20260925 02:59:58.744688 1741 log.cc:1196] T 279de86fe45949dd9d41bec861b483ee P e83773d3745a43149e1afdf48c90e0b1: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/wal/wals/279de86fe45949dd9d41bec861b483ee
I20260925 02:59:58.745239 1741 ts_tablet_manager.cc:1939] T 279de86fe45949dd9d41bec861b483ee P e83773d3745a43149e1afdf48c90e0b1: Deleting consensus metadata
I20260925 02:59:58.746230 1712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:58.723189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49082 (local address 127.18.196.193:32845)
0925 02:59:58.723357 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:58.723363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:58.724022 (+ 659us) server_negotiation.cc:407] Connection header received
0925 02:59:58.724081 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:58.724086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:58.724193 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:58.724285 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:58.726223 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.727449 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.729444 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.730079 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.730206 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:58.736465 (+ 6259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:58.736478 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:58.736494 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:58.736544 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:58.740533 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:58.741723 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:58.741732 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:58.741735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:58.741896 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:58.743237 (+ 1341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:58.743243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:58.743246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:58.743733 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:58.744016 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:58.745611 (+ 1595us) server_negotiation.cc:299] Negotiation successful
0925 02:59:58.745862 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:59:58.746510 1234 catalog_manager.cc:4841] TS 367161aa0d6441f186711e0eaef22188 (127.18.196.195:34925): tablet 279de86fe45949dd9d41bec861b483ee (table test-table [id=6e1c224c761e4ff594aa850ebd0a3132]) successfully deleted
I20260925 02:59:58.747507 1431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:59:58.747632 1234 catalog_manager.cc:4841] TS e83773d3745a43149e1afdf48c90e0b1 (127.18.196.193:32845): tablet 279de86fe45949dd9d41bec861b483ee (table test-table [id=6e1c224c761e4ff594aa850ebd0a3132]) successfully deleted
I20260925 02:59:58.755452 1742 tablet_replica.cc:330] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba: stopping tablet replica
I20260925 02:59:58.755929 1742 raft_consensus.cc:2229] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:59:58.756644 1742 raft_consensus.cc:2258] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:59:58.758695 1742 ts_tablet_manager.cc:1905] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:59:58.766865 1742 ts_tablet_manager.cc:1918] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:59:58.767028 1742 log.cc:1196] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/279de86fe45949dd9d41bec861b483ee
I20260925 02:59:58.767585 1742 ts_tablet_manager.cc:1939] T 279de86fe45949dd9d41bec861b483ee P 621383aa8000491bbdc36bf8de3e80ba: Deleting consensus metadata
I20260925 02:59:58.769251 1235 catalog_manager.cc:4841] TS 621383aa8000491bbdc36bf8de3e80ba (127.18.196.194:39683): tablet 279de86fe45949dd9d41bec861b483ee (table test-table [id=6e1c224c761e4ff594aa850ebd0a3132]) successfully deleted
I20260925 02:59:58.844080 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 1287
I20260925 02:59:58.863287 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 1440
I20260925 02:59:58.878604 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:58.864038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.18.196.194:39683)
0925 02:59:58.864231 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:58.864238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:58.864260 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:59:58.864380 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:58.864385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:58.864471 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:58.864593 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:58.865919 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.867229 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.868546 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:58.869236 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:58.869347 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:58.871895 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:58.871910 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:58.871929 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:58.871971 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:58.875831 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:58.876478 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:58.876488 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:58.876492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:58.876674 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:58.877002 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:58.877007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:58.877011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:58.877485 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:58.877800 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:58.878006 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 02:59:58.878225 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 02:59:58.879598 1564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:59:58.989192 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 1440
I20260925 02:59:59.008270 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 1572
I20260925 02:59:59.033654 1715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:59:59.013260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60386 (local address 127.18.196.195:34925)
0925 02:59:59.013432 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 02:59:59.013436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:59:59.013453 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:59:59.013635 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:59:59.013639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:59:59.013701 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:59:59.013803 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:59:59.015333 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:59.016153 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:59.017431 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:59:59.017832 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:59:59.017928 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:59:59.028622 (+ 10694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:59:59.028638 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:59:59.028653 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:59:59.028705 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:59:59.031310 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:59.031973 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:59.031979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:59.031982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:59.032103 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:59:59.032423 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:59:59.032426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:59:59.032428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:59:59.032766 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:59:59.032973 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 02:59:59.033230 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 02:59:59.033382 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:59:59.034271 1696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:59:59.122377 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 1572
I20260925 02:59:59.139240 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 1217
I20260925 02:59:59.140460 1280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:59:59.255133 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 1217
I20260925 02:59:59.276768 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:39615
--webserver_interface=127.18.196.254
--webserver_port=46265
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:37687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:39615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:59:59.607594 1748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:59:59.607955 1748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:59:59.608129 1748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:59:59.615422 1748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:59:59.615536 1748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:59:59.615624 1748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:59:59.615712 1748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:59:59.615782 1748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:59:59.625692 1748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:37687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:39615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:39615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=46265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:59:59.627070 1748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:59:59.628381 1748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:59:59.640327 1753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:59.640364 1756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:59:59.640292 1754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:59:59.641031 1748 server_base.cc:1034] running on GCE node
I20260925 02:59:59.641760 1748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:59:59.643352 1748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:59:59.644586 1748 hybrid_clock.cc:648] HybridClock initialized: now 1790305199644548 us; error 62 us; skew 500 ppm
I20260925 02:59:59.644956 1748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:59:59 dist-test-slave-8zvk krb5kdc[664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305199, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:59:59.648088 1748 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 02:59:59.649544 1748 webserver.cc:460] Webserver started at http://127.18.196.254:46265/ using document root <none> and password file <none>
I20260925 02:59:59.650203 1748 fs_manager.cc:359] Metadata directory not provided
I20260925 02:59:59.650350 1748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:59:59.655602 1748 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:59:59.658227 1763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:59:59.659502 1748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:59:59.659663 1748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "fb2916c744e440ec97ee79298c2eed96"
format_stamp: "Formatted at 2026-09-25 02:59:54 on dist-test-slave-8zvk"
server_key: "94a7defae33de466c27aae988bc5bbbc"
server_key_iv: "c6c94a417f02982cb0a44c2fab4f0bff"
server_key_version: "encryptionkey@0"
I20260925 02:59:59.660629 1748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:59:59.686163 1748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:59:59.690418 1748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:59:59.690706 1748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:59:59.709228 1748 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:39615
I20260925 02:59:59.709263 1814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:39615 every 8 connection(s)
I20260925 02:59:59.710680 1748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 02:59:59.717154 1815 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:59:59.720345 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 1748
I20260925 02:59:59.720595 1815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96: Bootstrap starting.
I20260925 02:59:59.721362 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:32845
--local_ip_for_outbound_sockets=127.18.196.193
--tserver_master_addrs=127.18.196.254:39615
--webserver_port=40759
--webserver_interface=127.18.196.193
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:37687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:59:59.743858 1815 log.cc:824] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96: Log is configured to *not* fsync() on all Append() calls
I20260925 02:59:59.759577 1815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:59:59.760128 1815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96: Bootstrap complete.
I20260925 02:59:59.769481 1815 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2916c744e440ec97ee79298c2eed96" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39615 } }
I20260925 02:59:59.770336 1815 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb2916c744e440ec97ee79298c2eed96, State: Initialized, Role: FOLLOWER
I20260925 02:59:59.771039 1815 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2916c744e440ec97ee79298c2eed96" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39615 } }
I20260925 02:59:59.771270 1815 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:59:59.771410 1815 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:59:59.771587 1815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 2 FOLLOWER]: Advancing to term 3
I20260925 02:59:59.775388 1815 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2916c744e440ec97ee79298c2eed96" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39615 } }
I20260925 02:59:59.775868 1815 leader_election.cc:302] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb2916c744e440ec97ee79298c2eed96; no voters:
I20260925 02:59:59.776516 1815 leader_election.cc:288] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 02:59:59.776660 1819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 02:59:59.777849 1819 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [term 3 LEADER]: Becoming Leader. State: Replica: fb2916c744e440ec97ee79298c2eed96, State: Running, Role: LEADER
I20260925 02:59:59.778429 1819 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2916c744e440ec97ee79298c2eed96" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39615 } }
I20260925 02:59:59.778726 1815 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:59:59.781188 1821 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb2916c744e440ec97ee79298c2eed96. Latest consensus state: current_term: 3 leader_uuid: "fb2916c744e440ec97ee79298c2eed96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2916c744e440ec97ee79298c2eed96" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39615 } } }
I20260925 02:59:59.781550 1821 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [sys.catalog]: This master's current role is: LEADER
I20260925 02:59:59.781268 1820 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fb2916c744e440ec97ee79298c2eed96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2916c744e440ec97ee79298c2eed96" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39615 } } }
I20260925 02:59:59.781688 1820 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2916c744e440ec97ee79298c2eed96 [sys.catalog]: This master's current role is: LEADER
I20260925 02:59:59.792059 1825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:59:59.802991 1825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:59:59.805116 1825 catalog_manager.cc:1259] Loaded cluster ID: e0ba09d6c9704269960ffab4fa1c5745
I20260925 02:59:59.805313 1825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 02:59:59.808005 1825 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f686ad469ad google::LogMessage::Fail() at ??:0
@ 0x7f686ad4988a google::LogMessage::SendToLog() at ??:0
@ 0x7f686ad464de google::LogMessage::Flush() at ??:0
@ 0x7f686ad47a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6880779512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f6880776976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f68807fe610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f68807fe439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f686d992345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f686d99591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f686d995799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f686d970eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6870c4a6db start_thread at ??:0
@ 0x7f6869be061f clone at ??:0
W20260925 03:00:00.137692 1817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:00:00.137961 1817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:00:00.138055 1817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:00:00.145149 1817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:00:00.145246 1817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:00:00.145351 1817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 03:00:00.154588 1817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:37687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:32845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=40759
--tserver_master_addrs=127.18.196.254:39615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:00:00.155908 1817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:00:00.157138 1817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:00:00.170542 1857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:00:00.170575 1859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:00:00.170828 1856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:00:00.171473 1817 server_base.cc:1034] running on GCE node
I20260925 03:00:00.172045 1817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:00:00.172923 1817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:00:00.174146 1817 hybrid_clock.cc:648] HybridClock initialized: now 1790305200174107 us; error 45 us; skew 500 ppm
I20260925 03:00:00.174431 1817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:00:00 dist-test-slave-8zvk krb5kdc[664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305200, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:00:00.177395 1817 init.cc:377] Logged in from keytab as kudu/127.18.196.193@KRBTEST.COM (short username kudu)
I20260925 03:00:00.178802 1817 webserver.cc:460] Webserver started at http://127.18.196.193:40759/ using document root <none> and password file <none>
I20260925 03:00:00.179493 1817 fs_manager.cc:359] Metadata directory not provided
I20260925 03:00:00.179642 1817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:00:00.184753 1817 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260925 03:00:00.187287 1866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:00.188563 1817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 03:00:00.188752 1817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "e83773d3745a43149e1afdf48c90e0b1"
format_stamp: "Formatted at 2026-09-25 02:59:54 on dist-test-slave-8zvk"
server_key: "7a4cf11b744cb2c6c3f4e0028439c259"
server_key_iv: "49b8041bf5ddbd544c14ba973d85c19f"
server_key_version: "encryptionkey@0"
I20260925 03:00:00.189718 1817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:00:00.220674 1817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:00:00.224411 1817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:00:00.224674 1817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:00:00.225610 1817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:00:00.226817 1817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:00:00.226958 1817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:00.227075 1817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:00:00.227159 1817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:00.260454 1817 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:32845
I20260925 03:00:00.260537 1978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:32845 every 8 connection(s)
I20260925 03:00:00.261912 1817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 03:00:00.266520 1981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:00.264828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.193:44849)
0925 03:00:00.265504 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:00.265721 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":274,"threads_started":1}
W20260925 03:00:00.267179 1979 heartbeater.cc:644] Failed to heartbeat to 127.18.196.254:39615 (0 consecutive failures): Network error: Failed to ping master at 127.18.196.254:39615: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
I20260925 03:00:00.267920 1981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:00.267620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.193:36381)
0925 03:00:00.267757 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:00.267820 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 03:00:00.268683 1981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:00.268386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.193:46617)
0925 03:00:00.268497 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:00.268581 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 03:00:00.268903 1979 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 03:00:00.269145 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 1817
I20260925 03:00:00.269814 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:39683
--local_ip_for_outbound_sockets=127.18.196.194
--tserver_master_addrs=127.18.196.254:39615
--webserver_port=35843
--webserver_interface=127.18.196.194
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:37687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:00:00.581876 1982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:00:00.582108 1982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:00:00.582275 1982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:00:00.589010 1982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:00:00.589107 1982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:00:00.589318 1982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 03:00:00.598425 1982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:37687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:39683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=35843
--tserver_master_addrs=127.18.196.254:39615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:00:00.599501 1982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:00:00.600716 1982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:00:00.613279 1990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:00:00.613358 1988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:00:00.613746 1987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:00:00.614092 1982 server_base.cc:1034] running on GCE node
I20260925 03:00:00.614688 1982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:00:00.615629 1982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:00:00.616873 1982 hybrid_clock.cc:648] HybridClock initialized: now 1790305200616850 us; error 58 us; skew 500 ppm
I20260925 03:00:00.617177 1982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:00:00 dist-test-slave-8zvk krb5kdc[664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305200, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:00:00.620253 1982 init.cc:377] Logged in from keytab as kudu/127.18.196.194@KRBTEST.COM (short username kudu)
I20260925 03:00:00.621565 1982 webserver.cc:460] Webserver started at http://127.18.196.194:35843/ using document root <none> and password file <none>
I20260925 03:00:00.622246 1982 fs_manager.cc:359] Metadata directory not provided
I20260925 03:00:00.622392 1982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:00:00.627607 1982 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 03:00:00.630189 1997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:00.631457 1982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 03:00:00.631632 1982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "621383aa8000491bbdc36bf8de3e80ba"
format_stamp: "Formatted at 2026-09-25 02:59:55 on dist-test-slave-8zvk"
server_key: "23bd3733b288b8508b3932600d3bcd0c"
server_key_iv: "57e29a9683bad97027e504a57c9a8e06"
server_key_version: "encryptionkey@0"
I20260925 03:00:00.632579 1982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:00:00.649523 1982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:00:00.653494 1982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:00:00.653776 1982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:00:00.654773 1982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:00:00.656067 1982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:00:00.656203 1982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:00.656343 1982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:00:00.656443 1982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:00.689517 1982 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:39683
I20260925 03:00:00.689600 2112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:39683 every 8 connection(s)
I20260925 03:00:00.691061 1982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 03:00:00.691444 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 1982
I20260925 03:00:00.692351 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:34925
--local_ip_for_outbound_sockets=127.18.196.195
--tserver_master_addrs=127.18.196.254:39615
--webserver_port=43035
--webserver_interface=127.18.196.195
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:37687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 03:00:00.696468 2115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:00.694442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.194:33887)
0925 03:00:00.695300 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:00.695549 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":363,"threads_started":1}
W20260925 03:00:00.697182 2113 heartbeater.cc:644] Failed to heartbeat to 127.18.196.254:39615 (0 consecutive failures): Network error: Failed to ping master at 127.18.196.254:39615: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
I20260925 03:00:00.698069 2115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:00.697739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.194:37147)
0925 03:00:00.697895 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:00.697969 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 03:00:00.699040 2115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:00.698587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.194:41705)
0925 03:00:00.698760 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:00.698883 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 03:00:00.699302 2113 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 03:00:01.015283 2116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:00:01.015578 2116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:00:01.015751 2116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:00:01.022877 2116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:00:01.023031 2116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:00:01.023202 2116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 03:00:01.032519 2116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:37687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:34925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=43035
--tserver_master_addrs=127.18.196.254:39615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:00:01.033780 2116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:00:01.035158 2116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:00:01.047668 2121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:00:01.047726 2122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:00:01.047761 2124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:00:01.048946 2116 server_base.cc:1034] running on GCE node
I20260925 03:00:01.049525 2116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:00:01.050367 2116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:00:01.051564 2116 hybrid_clock.cc:648] HybridClock initialized: now 1790305201051527 us; error 25 us; skew 500 ppm
I20260925 03:00:01.051883 2116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:00:01 dist-test-slave-8zvk krb5kdc[664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305201, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:00:01.054816 2116 init.cc:377] Logged in from keytab as kudu/127.18.196.195@KRBTEST.COM (short username kudu)
I20260925 03:00:01.056203 2116 webserver.cc:460] Webserver started at http://127.18.196.195:43035/ using document root <none> and password file <none>
I20260925 03:00:01.056856 2116 fs_manager.cc:359] Metadata directory not provided
I20260925 03:00:01.056998 2116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:00:01.061890 2116 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.001s
I20260925 03:00:01.064436 2131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:01.065687 2116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 03:00:01.065855 2116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "367161aa0d6441f186711e0eaef22188"
format_stamp: "Formatted at 2026-09-25 02:59:55 on dist-test-slave-8zvk"
server_key: "90ba3374d54814a7246ca94e4c7b10ff"
server_key_iv: "161c30ac7d9e0a670a476c38dc282cc8"
server_key_version: "encryptionkey@0"
I20260925 03:00:01.066787 2116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:00:01.081305 2116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:00:01.085194 2116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:00:01.085475 2116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:00:01.086395 2116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:00:01.087754 2116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:00:01.087881 2116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:01.088016 2116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:00:01.088110 2116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:01.121321 2116 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:34925
I20260925 03:00:01.121402 2244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:34925 every 8 connection(s)
I20260925 03:00:01.122788 2116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 03:00:01.125983 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 2116
I20260925 03:00:01.127683 2247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:01.125490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.195:48947)
0925 03:00:01.126341 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:01.126623 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":300,"threads_started":1}
W20260925 03:00:01.128324 2245 heartbeater.cc:644] Failed to heartbeat to 127.18.196.254:39615 (0 consecutive failures): Network error: Failed to ping master at 127.18.196.254:39615: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
I20260925 03:00:01.128988 2247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:01.128706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.195:45029)
0925 03:00:01.128832 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:01.128893 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 03:00:01.129966 2247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:01.129588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.195:46197)
0925 03:00:01.129751 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:01.129855 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260925 03:00:01.130213 2245 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 03:00:01.270321 2249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:01.269439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.193:40325)
0925 03:00:01.269948 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:01.270091 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":251,"threads_started":1}
I20260925 03:00:01.700525 2250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:01.699779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.194:44283)
0925 03:00:01.700193 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:01.700314 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":203,"threads_started":1}
I20260925 03:00:02.131491 2251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:02.130731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.195:45105)
0925 03:00:02.131166 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:02.131294 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":235,"threads_started":1}
I20260925 03:00:02.271963 2252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:02.271292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.193:36943)
0925 03:00:02.271718 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:02.271800 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":204,"threads_started":1}
I20260925 03:00:02.702328 2253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:02.701515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.194:41997)
0925 03:00:02.702074 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:02.702153 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":312,"threads_started":1}
I20260925 03:00:03.132918 2254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:03.132220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.195:51793)
0925 03:00:03.132699 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:03.132774 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":269,"threads_started":1}
I20260925 03:00:03.273808 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:03.272925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.193:43311)
0925 03:00:03.273463 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:03.273557 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":281,"threads_started":1}
I20260925 03:00:03.703966 2257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:03.703279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.194:42861)
0925 03:00:03.703710 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:03.703796 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":216,"threads_started":1}
I20260925 03:00:04.134361 2258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:04.133700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.195:35207)
0925 03:00:04.134118 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:04.134206 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":228,"threads_started":1}
I20260925 03:00:04.275486 2259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:04.274773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.193:42065)
0925 03:00:04.275230 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:04.275320 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":242,"threads_started":1}
I20260925 03:00:04.705715 2260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:04.704958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.194:49739)
0925 03:00:04.705429 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:04.705518 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":235,"threads_started":1}
I20260925 03:00:05.136029 2261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:05.135297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.195:34285)
0925 03:00:05.135732 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:05.135827 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":208,"threads_started":1}
I20260925 03:00:05.277046 2262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:05.276362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.193:38873)
0925 03:00:05.276795 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:05.276877 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":214,"threads_started":1}
I20260925 03:00:05.707510 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:05.706607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.194:39941)
0925 03:00:05.707190 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:05.707275 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":330,"threads_started":1}
I20260925 03:00:06.137503 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:06.136787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.195:51801)
0925 03:00:06.137275 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:06.137350 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":279,"threads_started":1}
I20260925 03:00:06.278584 2265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:06.277913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.193:34263)
0925 03:00:06.278343 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:06.278426 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":229,"threads_started":1}
I20260925 03:00:06.709067 2266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:06.708369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.194:60439)
0925 03:00:06.708822 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:06.708909 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":242,"threads_started":1}
I20260925 03:00:07.139083 2267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:07.138307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.195:50697)
0925 03:00:07.138774 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:07.138926 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":248,"threads_started":1}
I20260925 03:00:07.280148 2268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:07.279432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.193:51829)
0925 03:00:07.279878 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:07.279957 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":202,"threads_started":1}
I20260925 03:00:07.710614 2269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:07.709870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.194:53741)
0925 03:00:07.710352 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:07.710422 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":209,"threads_started":1}
I20260925 03:00:08.141036 2270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:08.139989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.195:51029)
0925 03:00:08.140628 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:08.140746 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":272,"threads_started":1}
I20260925 03:00:08.282253 2271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:08.281248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.193:49137)
0925 03:00:08.281883 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:08.281998 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":346,"threads_started":1}
I20260925 03:00:08.712146 2272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:08.711430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.194:38847)
0925 03:00:08.711850 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:08.711929 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":224,"threads_started":1}
I20260925 03:00:09.142763 2273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:09.142015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.195:39427)
0925 03:00:09.142505 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:09.142589 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":283,"threads_started":1}
I20260925 03:00:09.283870 2274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:09.283119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.193:56513)
0925 03:00:09.283598 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:09.283704 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":235,"threads_started":1}
I20260925 03:00:09.713681 2275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:09.712998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.194:49177)
0925 03:00:09.713446 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:09.713521 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":217,"threads_started":1}
I20260925 03:00:10.144305 2276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:10.143627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.195:33373)
0925 03:00:10.144058 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:10.144132 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":216,"threads_started":1}
I20260925 03:00:10.285504 2277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:10.284827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.193:42151)
0925 03:00:10.285272 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:10.285350 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":242,"threads_started":1}
I20260925 03:00:10.715204 2278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:10.714507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.194:35341)
0925 03:00:10.714983 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:10.715059 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":248,"threads_started":1}
I20260925 03:00:11.145774 2279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:11.145126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.195:40297)
0925 03:00:11.145543 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:11.145617 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":234,"threads_started":1}
I20260925 03:00:11.287323 2280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:11.286458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.193:57237)
0925 03:00:11.286989 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:11.287079 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":275,"threads_started":1}
I20260925 03:00:11.716799 2281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:11.716022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.194:51165)
0925 03:00:11.716527 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:11.716612 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":302,"threads_started":1}
I20260925 03:00:12.147432 2282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:12.146650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.195:36207)
0925 03:00:12.147165 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:12.147258 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":295,"threads_started":1}
I20260925 03:00:12.288880 2283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:12.288217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.193:56335)
0925 03:00:12.288631 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:12.288717 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":210,"threads_started":1}
I20260925 03:00:12.718333 2284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:12.717675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.194:41491)
0925 03:00:12.718099 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:12.718189 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":215,"threads_started":1}
I20260925 03:00:13.148864 2285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:13.148243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.195:37259)
0925 03:00:13.148625 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:13.148698 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":204,"threads_started":1}
I20260925 03:00:13.290823 2286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:13.289843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.193:59505)
0925 03:00:13.290454 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:13.290586 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":331,"threads_started":1}
I20260925 03:00:13.720114 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:13.719389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.194:39301)
0925 03:00:13.719846 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:13.719933 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":236,"threads_started":1}
I20260925 03:00:14.150430 2288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:14.149713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.195:41325)
0925 03:00:14.150158 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:14.150256 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":218,"threads_started":1}
I20260925 03:00:14.292446 2289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:14.291697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.193:47463)
0925 03:00:14.292123 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:14.292203 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":211,"threads_started":1}
I20260925 03:00:14.721916 2290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:14.721177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.194:34861)
0925 03:00:14.721640 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:14.721733 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":221,"threads_started":1}
I20260925 03:00:15.152299 2291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:15.151607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.195:38845)
0925 03:00:15.152017 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:15.152105 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":204,"threads_started":1}
I20260925 03:00:15.294770 2292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:15.293733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.193:40309)
0925 03:00:15.294363 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:15.294502 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":272,"threads_started":1}
I20260925 03:00:15.723757 2293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:15.723041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.194:35767)
0925 03:00:15.723463 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:15.723554 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":210,"threads_started":1}
I20260925 03:00:16.132640 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 1817
I20260925 03:00:16.150545 2294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:16.133729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54984 (local address 127.18.196.193:32845)
0925 03:00:16.134482 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:16.134497 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:16.134534 (+ 37us) server_negotiation.cc:407] Connection header received
0925 03:00:16.136637 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:16.136646 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:16.137040 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:16.137329 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:00:16.138747 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:16.140008 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:16.141436 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:16.142076 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:16.142219 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:16.144737 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:00:16.144746 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:00:16.144769 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:00:16.144818 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:00:16.147603 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:16.148227 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:16.148238 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:16.148244 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:16.148398 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:16.148736 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:16.148740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:16.148742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:16.149268 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:00:16.149502 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:16.149870 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 03:00:16.150272 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":378,"threads_started":1}
I20260925 03:00:16.151916 1974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:00:16.154373 2296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:16.153432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39615 (local address 127.18.196.195:36155)
0925 03:00:16.153953 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:16.154095 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":230,"threads_started":1}
I20260925 03:00:16.233942 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 1817
I20260925 03:00:16.251070 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 1982
I20260925 03:00:16.275573 2297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:16.252074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45166 (local address 127.18.196.194:39683)
0925 03:00:16.252585 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:16.252599 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:16.252635 (+ 36us) server_negotiation.cc:407] Connection header received
0925 03:00:16.254115 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:16.254124 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:16.254526 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:16.254824 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:00:16.256211 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:16.257431 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:16.258599 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:16.259337 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:16.259461 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:16.269379 (+ 9918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:00:16.269387 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:00:16.269402 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:00:16.269445 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:00:16.272741 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:16.273389 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:16.273395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:16.273401 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:16.273561 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:16.273874 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:16.273877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:16.273879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:16.274348 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:00:16.274562 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:16.274974 (+ 412us) server_negotiation.cc:299] Negotiation successful
0925 03:00:16.275282 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":226,"threads_started":1}
I20260925 03:00:16.276957 2108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:00:16.359061 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 1982
I20260925 03:00:16.377556 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 2116
I20260925 03:00:16.394186 2299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:16.378533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55382 (local address 127.18.196.195:34925)
0925 03:00:16.379135 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:16.379149 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:16.379193 (+ 44us) server_negotiation.cc:407] Connection header received
0925 03:00:16.380855 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:16.380866 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:16.381272 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:16.381549 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:00:16.382987 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:16.384218 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:16.385428 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:16.386058 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:16.386185 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:16.388646 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:00:16.388653 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:00:16.388666 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:00:16.388715 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:00:16.391440 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:16.392038 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:16.392044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:16.392050 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:16.392214 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:16.392528 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:16.392531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:16.392533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:16.393025 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:00:16.393256 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:16.393570 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 03:00:16.393878 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":291,"threads_started":1}
I20260925 03:00:16.395529 2240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:00:16.479197 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 2116
2026-09-25T03:00:16Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24110 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 03:00:16 dist-test-slave-8zvk krb5kdc[2303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 03:00:16 dist-test-slave-8zvk krb5kdc[2303](info): set up 2 sockets
Sep 25 03:00:16 dist-test-slave-8zvk krb5kdc[2303](info): commencing operation
krb5kdc: starting...
W20260925 03:00:17.562702 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.004s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 03:00:17 dist-test-slave-8zvk krb5kdc[2303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T03:00:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:00:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 03:00:17.790460 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:46387
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:46387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:00:18.103096 2316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:00:18.103402 2316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:00:18.103566 2316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:00:18.110409 2316 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 03:00:18.110546 2316 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 03:00:18.110639 2316 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 03:00:18.110725 2316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:00:18.110803 2316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:00:18.110918 2316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:00:18.111006 2316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:00:18.111099 2316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:00:18.120345 2316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:46387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:46387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:00:18.121519 2316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:00:18.122804 2316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:00:18.134299 2324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:00:18.134384 2322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:00:18.134333 2321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:00:18.135085 2316 server_base.cc:1034] running on GCE node
I20260925 03:00:18.135772 2316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:00:18.136982 2316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:00:18.138217 2316 hybrid_clock.cc:648] HybridClock initialized: now 1790305218138153 us; error 45 us; skew 500 ppm
I20260925 03:00:18.138523 2316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:00:18 dist-test-slave-8zvk krb5kdc[2303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305218, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:00:18.141589 2316 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 03:00:18.142915 2316 webserver.cc:460] Webserver started at http://127.18.196.254:35321/ using document root <none> and password file <none>
I20260925 03:00:18.143525 2316 fs_manager.cc:359] Metadata directory not provided
I20260925 03:00:18.143606 2316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:00:18.143857 2316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:00:18.146708 2316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "89ae4c9f9042478499d8c6a40f3cc0bb"
format_stamp: "Formatted at 2026-09-25 03:00:18 on dist-test-slave-8zvk"
server_key: "880f2ee6192cb43171778870400ebfd1"
server_key_iv: "ceb103f29afb2173dde1e580916dca46"
server_key_version: "encryptionkey@0"
I20260925 03:00:18.147480 2316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "89ae4c9f9042478499d8c6a40f3cc0bb"
format_stamp: "Formatted at 2026-09-25 03:00:18 on dist-test-slave-8zvk"
server_key: "880f2ee6192cb43171778870400ebfd1"
server_key_iv: "ceb103f29afb2173dde1e580916dca46"
server_key_version: "encryptionkey@0"
I20260925 03:00:18.152338 2316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 03:00:18.155623 2331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:18.156873 2316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 03:00:18.157037 2316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "89ae4c9f9042478499d8c6a40f3cc0bb"
format_stamp: "Formatted at 2026-09-25 03:00:18 on dist-test-slave-8zvk"
server_key: "880f2ee6192cb43171778870400ebfd1"
server_key_iv: "ceb103f29afb2173dde1e580916dca46"
server_key_version: "encryptionkey@0"
I20260925 03:00:18.157236 2316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:00:18.168243 2316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:00:18.171844 2316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:00:18.172094 2316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:00:18.189481 2316 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:46387
I20260925 03:00:18.189500 2382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:46387 every 8 connection(s)
I20260925 03:00:18.190737 2316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:00:18.194168 2383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:00:18.200898 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 2316
I20260925 03:00:18.201102 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:00:18.201750 19219 external_mini_cluster.cc:1428] Setting key a22504cc33069e1b5b5da25a6a2495fb
I20260925 03:00:18.204643 2383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb: Bootstrap starting.
I20260925 03:00:18.208600 2383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb: Neither blocks nor log segments found. Creating new log.
I20260925 03:00:18.209703 2383 log.cc:824] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb: Log is configured to *not* fsync() on all Append() calls
I20260925 03:00:18.212482 2383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb: No bootstrap required, opened a new log
Sep 25 03:00:18 dist-test-slave-8zvk krb5kdc[2303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:00:18.221906 2383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae4c9f9042478499d8c6a40f3cc0bb" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 46387 } }
I20260925 03:00:18.222322 2383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:00:18.222426 2383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89ae4c9f9042478499d8c6a40f3cc0bb, State: Initialized, Role: FOLLOWER
I20260925 03:00:18.223107 2383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae4c9f9042478499d8c6a40f3cc0bb" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 46387 } }
I20260925 03:00:18.223402 2383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:00:18.223554 2383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:00:18.223738 2383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:00:18.227118 2383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae4c9f9042478499d8c6a40f3cc0bb" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 46387 } }
I20260925 03:00:18.227435 2386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:18.203851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41440 (local address 127.18.196.254:46387)
0925 03:00:18.204649 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:18.204666 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:18.204719 (+ 53us) server_negotiation.cc:407] Connection header received
0925 03:00:18.207171 (+ 2452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:18.207210 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:18.207653 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:18.208072 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:00:18.209457 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:18.211305 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:18.212643 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:18.213704 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:18.213879 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:18.217826 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:00:18.217836 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:00:18.217854 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:00:18.217896 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:00:18.221843 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:18.222648 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:18.222656 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:18.222666 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:18.222893 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:18.223264 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:18.223269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:18.223272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:18.223998 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:00:18.224311 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:18.225665 (+ 1354us) server_negotiation.cc:299] Negotiation successful
0925 03:00:18.226099 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":367,"threads_started":1}
I20260925 03:00:18.227677 2383 leader_election.cc:302] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89ae4c9f9042478499d8c6a40f3cc0bb; no voters:
I20260925 03:00:18.228348 2383 leader_election.cc:288] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:00:18.228488 2388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:00:18.230216 2383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:00:18.230918 2388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 1 LEADER]: Becoming Leader. State: Replica: 89ae4c9f9042478499d8c6a40f3cc0bb, State: Running, Role: LEADER
I20260925 03:00:18.231480 2388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae4c9f9042478499d8c6a40f3cc0bb" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 46387 } }
I20260925 03:00:18.234910 2389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89ae4c9f9042478499d8c6a40f3cc0bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae4c9f9042478499d8c6a40f3cc0bb" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 46387 } } }
I20260925 03:00:18.234980 2390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89ae4c9f9042478499d8c6a40f3cc0bb. Latest consensus state: current_term: 1 leader_uuid: "89ae4c9f9042478499d8c6a40f3cc0bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae4c9f9042478499d8c6a40f3cc0bb" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 46387 } } }
I20260925 03:00:18.235370 2390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [sys.catalog]: This master's current role is: LEADER
I20260925 03:00:18.235378 2389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [sys.catalog]: This master's current role is: LEADER
I20260925 03:00:18.236968 2398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:00:18.241305 2398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:00:18.249765 2398 catalog_manager.cc:1347] Generated new cluster ID: fb3e3f55110242f18de7b693acb25141
I20260925 03:00:18.249902 2398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:00:18.266247 2398 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:00:18.267590 2398 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:00:18.277752 2398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb: Generated new TSK 0
I20260925 03:00:18.278561 2398 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
I20260925 03:00:18.344553 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:46387
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:00:18.667366 2410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:00:18.667618 2410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:00:18.667783 2410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:00:18.674952 2410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:00:18.675093 2410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:00:18.675258 2410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 03:00:18.684760 2410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:46387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:00:18.686228 2410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:00:18.687532 2410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:00:18.700196 2416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:00:18.700253 2418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:00:18.700354 2415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:00:18.701292 2410 server_base.cc:1034] running on GCE node
I20260925 03:00:18.701896 2410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:00:18.702819 2410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:00:18.704101 2410 hybrid_clock.cc:648] HybridClock initialized: now 1790305218704053 us; error 40 us; skew 500 ppm
I20260925 03:00:18.704425 2410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:00:18 dist-test-slave-8zvk krb5kdc[2303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305218, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:00:18.707554 2410 init.cc:377] Logged in from keytab as kudu/127.18.196.193@KRBTEST.COM (short username kudu)
I20260925 03:00:18.708972 2410 webserver.cc:460] Webserver started at http://127.18.196.193:43231/ using document root <none> and password file <none>
I20260925 03:00:18.709625 2410 fs_manager.cc:359] Metadata directory not provided
I20260925 03:00:18.709769 2410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:00:18.710080 2410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:00:18.713066 2410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "a339037fb3c14611b0ca6c8892c83e87"
format_stamp: "Formatted at 2026-09-25 03:00:18 on dist-test-slave-8zvk"
server_key: "aefd6e2d7f6c33e9f1bd04ffad4905c4"
server_key_iv: "bab59001ce0220117fe1ed39b6094f19"
server_key_version: "encryptionkey@0"
I20260925 03:00:18.713852 2410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "a339037fb3c14611b0ca6c8892c83e87"
format_stamp: "Formatted at 2026-09-25 03:00:18 on dist-test-slave-8zvk"
server_key: "aefd6e2d7f6c33e9f1bd04ffad4905c4"
server_key_iv: "bab59001ce0220117fe1ed39b6094f19"
server_key_version: "encryptionkey@0"
I20260925 03:00:18.718778 2410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 03:00:18.722131 2425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:18.723415 2410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 03:00:18.723584 2410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "a339037fb3c14611b0ca6c8892c83e87"
format_stamp: "Formatted at 2026-09-25 03:00:18 on dist-test-slave-8zvk"
server_key: "aefd6e2d7f6c33e9f1bd04ffad4905c4"
server_key_iv: "bab59001ce0220117fe1ed39b6094f19"
server_key_version: "encryptionkey@0"
I20260925 03:00:18.723796 2410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:00:18.781199 2410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:00:18.785187 2410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:00:18.785485 2410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:00:18.786494 2410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:00:18.787900 2410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:00:18.788004 2410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:18.788148 2410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:00:18.788221 2410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:18.820993 2410 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:33491
I20260925 03:00:18.821044 2539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:33491 every 8 connection(s)
I20260925 03:00:18.822285 2410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 03:00:18.827481 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 2410
I20260925 03:00:18.827705 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 03:00:18.828529 19219 external_mini_cluster.cc:1428] Setting key 84d74407554619c3db972ed587632fee
Sep 25 03:00:18 dist-test-slave-8zvk krb5kdc[2303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305218, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:00:18.843169 2542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:18.824821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:35301 (local address 127.18.196.254:46387)
0925 03:00:18.825286 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:18.825290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:18.826065 (+ 775us) server_negotiation.cc:407] Connection header received
0925 03:00:18.828487 (+ 2422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:18.828492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:18.828559 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:18.828687 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:00:18.830717 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:18.831564 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:18.833115 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:18.833515 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:18.833648 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:18.837298 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:00:18.837306 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:00:18.837313 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:00:18.837374 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:00:18.840097 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:18.840955 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:18.840958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:18.840960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:18.841073 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:18.841497 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:18.841500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:18.841502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:18.841759 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:00:18.841865 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:18.842690 (+ 825us) server_negotiation.cc:299] Negotiation successful
0925 03:00:18.842911 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":259,"threads_started":1}
I20260925 03:00:18.844571 2543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:18.825270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.193:35301)
0925 03:00:18.825922 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:18.825957 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 03:00:18.828148 (+ 2191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:00:18.829043 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:00:18.829051 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:00:18.829754 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:00:18.830545 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:18.830582 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:18.831717 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:18.831723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:18.832970 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:18.832981 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:18.833749 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:18.833758 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:18.834088 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:18.834815 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:00:18.834886 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:00:18.837087 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:00:18.840231 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:00:18.840242 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:00:18.840258 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:00:18.840853 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:00:18.841170 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:00:18.841177 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:00:18.841183 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:00:18.841400 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:00:18.841894 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:00:18.841904 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:00:18.842317 (+ 413us) client_negotiation.cc:769] Sending connection context
0925 03:00:18.842605 (+ 288us) client_negotiation.cc:240] Negotiation successful
0925 03:00:18.843200 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":254,"threads_started":1}
I20260925 03:00:18.846139 2540 heartbeater.cc:347] Connected to a master server at 127.18.196.254:46387
I20260925 03:00:18.846451 2540 heartbeater.cc:464] Registering TS with master...
I20260925 03:00:18.847224 2540 heartbeater.cc:511] Master 127.18.196.254:46387 requested a full tablet report, sending...
I20260925 03:00:18.849275 2348 ts_manager.cc:194] Registered new tserver with Master: a339037fb3c14611b0ca6c8892c83e87 (127.18.196.193:33491)
I20260925 03:00:18.850700 2348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.193@KRBTEST.COM'} at 127.18.196.193:35301
WARNING: no policy specified for kudu/127.18.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
I20260925 03:00:18.883492 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:46387
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:00:19.208685 2547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:00:19.208985 2547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:00:19.209110 2547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:00:19.217455 2547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:00:19.217579 2547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:00:19.217775 2547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 03:00:19.229074 2547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:46387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:00:19.230280 2547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:00:19.231593 2547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:00:19.244382 2552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:00:19.244431 2553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:00:19.244386 2555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:00:19.245257 2547 server_base.cc:1034] running on GCE node
I20260925 03:00:19.245867 2547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:00:19.246742 2547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:00:19.248006 2547 hybrid_clock.cc:648] HybridClock initialized: now 1790305219247956 us; error 43 us; skew 500 ppm
I20260925 03:00:19.248306 2547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:00:19 dist-test-slave-8zvk krb5kdc[2303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305219, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:00:19.251273 2547 init.cc:377] Logged in from keytab as kudu/127.18.196.194@KRBTEST.COM (short username kudu)
I20260925 03:00:19.252586 2547 webserver.cc:460] Webserver started at http://127.18.196.194:33743/ using document root <none> and password file <none>
I20260925 03:00:19.253266 2547 fs_manager.cc:359] Metadata directory not provided
I20260925 03:00:19.253405 2547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:00:19.253785 2547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:00:19.256719 2547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "f7362975190748bd837bc87ec6f9e5ae"
format_stamp: "Formatted at 2026-09-25 03:00:19 on dist-test-slave-8zvk"
server_key: "623b0f2782b866c6d9a49f0a302d0c57"
server_key_iv: "c9826eebbf8b47ede715b069130ec636"
server_key_version: "encryptionkey@0"
I20260925 03:00:19.257467 2547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "f7362975190748bd837bc87ec6f9e5ae"
format_stamp: "Formatted at 2026-09-25 03:00:19 on dist-test-slave-8zvk"
server_key: "623b0f2782b866c6d9a49f0a302d0c57"
server_key_iv: "c9826eebbf8b47ede715b069130ec636"
server_key_version: "encryptionkey@0"
I20260925 03:00:19.262610 2547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 03:00:19.265986 2562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:19.267263 2547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 03:00:19.267424 2547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "f7362975190748bd837bc87ec6f9e5ae"
format_stamp: "Formatted at 2026-09-25 03:00:19 on dist-test-slave-8zvk"
server_key: "623b0f2782b866c6d9a49f0a302d0c57"
server_key_iv: "c9826eebbf8b47ede715b069130ec636"
server_key_version: "encryptionkey@0"
I20260925 03:00:19.267608 2547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:00:19.286557 2547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:00:19.289974 2547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:00:19.290243 2547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:00:19.291167 2547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:00:19.292572 2547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:00:19.292680 2547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:19.292824 2547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:00:19.292892 2547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:19.325351 2547 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:37227
I20260925 03:00:19.325426 2674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:37227 every 8 connection(s)
I20260925 03:00:19.326556 2547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 03:00:19.335167 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 2547
I20260925 03:00:19.335371 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 03:00:19.335836 19219 external_mini_cluster.cc:1428] Setting key 4811250da8924cecf38eb5201a07267d
Sep 25 03:00:19 dist-test-slave-8zvk krb5kdc[2303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305219, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:00:19.346366 2542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:19.329125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:51367 (local address 127.18.196.254:46387)
0925 03:00:19.329328 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:19.329332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:19.330354 (+ 1022us) server_negotiation.cc:407] Connection header received
0925 03:00:19.332294 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:19.332300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:19.332387 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:19.332518 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:00:19.333900 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.334715 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:19.336387 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.336900 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:19.337031 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:19.340669 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:00:19.340673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:00:19.340676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:00:19.340709 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:00:19.343332 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:19.344050 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:19.344055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:19.344057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:19.344185 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:19.344632 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:19.344636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:19.344638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:19.344899 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:00:19.345060 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:19.345946 (+ 886us) server_negotiation.cc:299] Negotiation successful
0925 03:00:19.346127 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 03:00:19.347880 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:19.329543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.194:51367)
0925 03:00:19.330199 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:19.330239 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 03:00:19.332022 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:00:19.332740 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:00:19.332747 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:00:19.333214 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:00:19.333729 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:19.333746 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.334880 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:19.334888 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:19.336236 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:19.336248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.337131 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:19.337137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:19.337344 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:19.338082 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:00:19.338105 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:00:19.340434 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:00:19.343468 (+ 3034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:00:19.343476 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:00:19.343492 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:00:19.343914 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:00:19.344306 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:00:19.344311 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:00:19.344315 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:00:19.344504 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:00:19.345029 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:00:19.345036 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:00:19.345462 (+ 426us) client_negotiation.cc:769] Sending connection context
0925 03:00:19.345781 (+ 319us) client_negotiation.cc:240] Negotiation successful
0925 03:00:19.346360 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":253,"threads_started":1}
I20260925 03:00:19.349380 2675 heartbeater.cc:347] Connected to a master server at 127.18.196.254:46387
I20260925 03:00:19.349723 2675 heartbeater.cc:464] Registering TS with master...
I20260925 03:00:19.350421 2675 heartbeater.cc:511] Master 127.18.196.254:46387 requested a full tablet report, sending...
I20260925 03:00:19.351879 2348 ts_manager.cc:194] Registered new tserver with Master: f7362975190748bd837bc87ec6f9e5ae (127.18.196.194:37227)
I20260925 03:00:19.352690 2348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.194@KRBTEST.COM'} at 127.18.196.194:51367
WARNING: no policy specified for kudu/127.18.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
I20260925 03:00:19.390349 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:46387
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:00:19.707202 2680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:00:19.707494 2680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:00:19.707661 2680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:00:19.714569 2680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:00:19.714708 2680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:00:19.714941 2680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 03:00:19.724638 2680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:46387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:00:19.725791 2680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:00:19.727089 2680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:00:19.740160 2685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:00:19.740224 2688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:00:19.740264 2686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:00:19.741420 2680 server_base.cc:1034] running on GCE node
I20260925 03:00:19.742079 2680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:00:19.743021 2680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:00:19.744225 2680 hybrid_clock.cc:648] HybridClock initialized: now 1790305219744176 us; error 37 us; skew 500 ppm
I20260925 03:00:19.744532 2680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:00:19 dist-test-slave-8zvk krb5kdc[2303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305219, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:00:19.747529 2680 init.cc:377] Logged in from keytab as kudu/127.18.196.195@KRBTEST.COM (short username kudu)
I20260925 03:00:19.748994 2680 webserver.cc:460] Webserver started at http://127.18.196.195:40497/ using document root <none> and password file <none>
I20260925 03:00:19.749686 2680 fs_manager.cc:359] Metadata directory not provided
I20260925 03:00:19.749815 2680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:00:19.750121 2680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:00:19.752966 2680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "3b5c6982fd6549ddb53e9a3aef215f1e"
format_stamp: "Formatted at 2026-09-25 03:00:19 on dist-test-slave-8zvk"
server_key: "c3d41831a73b82a666f4636dbef811bb"
server_key_iv: "6c213b760fb3411753b4f2f5602f6a17"
server_key_version: "encryptionkey@0"
I20260925 03:00:19.753738 2680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "3b5c6982fd6549ddb53e9a3aef215f1e"
format_stamp: "Formatted at 2026-09-25 03:00:19 on dist-test-slave-8zvk"
server_key: "c3d41831a73b82a666f4636dbef811bb"
server_key_iv: "6c213b760fb3411753b4f2f5602f6a17"
server_key_version: "encryptionkey@0"
I20260925 03:00:19.758728 2680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 03:00:19.762240 2695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:19.763722 2680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 03:00:19.763898 2680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "3b5c6982fd6549ddb53e9a3aef215f1e"
format_stamp: "Formatted at 2026-09-25 03:00:19 on dist-test-slave-8zvk"
server_key: "c3d41831a73b82a666f4636dbef811bb"
server_key_iv: "6c213b760fb3411753b4f2f5602f6a17"
server_key_version: "encryptionkey@0"
I20260925 03:00:19.764098 2680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:00:19.779978 2680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:00:19.783949 2680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:00:19.784245 2680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:00:19.785228 2680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:00:19.786581 2680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:00:19.786705 2680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:19.786903 2680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:00:19.786984 2680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260925 03:00:19.820360 2680 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:37589
I20260925 03:00:19.820416 2808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:37589 every 8 connection(s)
I20260925 03:00:19.822119 2680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 03:00:19.830984 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 2680
I20260925 03:00:19.831183 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 03:00:19.831789 19219 external_mini_cluster.cc:1428] Setting key e9fe321b8d11a88c4cde494794d23b91
Sep 25 03:00:19 dist-test-slave-8zvk krb5kdc[2303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305219, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:00:19.842590 2542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:19.824671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:39957 (local address 127.18.196.254:46387)
0925 03:00:19.824885 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:19.824892 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:19.826133 (+ 1241us) server_negotiation.cc:407] Connection header received
0925 03:00:19.828123 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:19.828129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:19.828284 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:19.828435 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:00:19.829848 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.830607 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:19.832787 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.833315 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:19.833457 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:19.837146 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:00:19.837156 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:00:19.837159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:00:19.837192 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:00:19.839654 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:19.840362 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:19.840366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:19.840368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:19.840493 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:19.840923 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:19.840928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:19.840930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:19.841195 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:00:19.841351 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:19.842028 (+ 677us) server_negotiation.cc:299] Negotiation successful
0925 03:00:19.842246 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 03:00:19.843591 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:19.825126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.195:39957)
0925 03:00:19.825948 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:19.825998 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 03:00:19.827834 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:00:19.828679 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:00:19.828687 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:00:19.829162 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:00:19.829705 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:19.829722 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.830777 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:19.830783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:19.832646 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:19.832659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.833548 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:19.833557 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:19.833883 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:19.834630 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:00:19.834655 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:00:19.836937 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:00:19.839802 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:00:19.839811 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:00:19.839827 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:00:19.840231 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:00:19.840603 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:00:19.840609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:00:19.840613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:00:19.840799 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:00:19.841321 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:00:19.841329 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:00:19.841663 (+ 334us) client_negotiation.cc:769] Sending connection context
0925 03:00:19.841914 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 03:00:19.842435 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":330,"threads_started":1}
I20260925 03:00:19.844868 2809 heartbeater.cc:347] Connected to a master server at 127.18.196.254:46387
I20260925 03:00:19.845194 2809 heartbeater.cc:464] Registering TS with master...
I20260925 03:00:19.845990 2809 heartbeater.cc:511] Master 127.18.196.254:46387 requested a full tablet report, sending...
I20260925 03:00:19.847572 2348 ts_manager.cc:194] Registered new tserver with Master: 3b5c6982fd6549ddb53e9a3aef215f1e (127.18.196.195:37589)
I20260925 03:00:19.848387 2348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.195@KRBTEST.COM'} at 127.18.196.195:39957
I20260925 03:00:19.853545 2540 heartbeater.cc:503] Master 127.18.196.254:46387 was elected leader, sending a full tablet report...
I20260925 03:00:19.859341 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 03:00:19.877271 2542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:19.863967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.18.196.254:46387)
0925 03:00:19.864149 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:19.864153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:19.864630 (+ 477us) server_negotiation.cc:407] Connection header received
0925 03:00:19.864799 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:19.864802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:19.864862 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:19.864944 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:00:19.866420 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.867295 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:19.868813 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.869222 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:19.869332 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:19.871882 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:00:19.871888 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:00:19.871891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:00:19.871935 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:00:19.874920 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:19.875577 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:19.875581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:19.875584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:19.875745 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:19.876063 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:19.876067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:19.876069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:19.876385 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:00:19.876572 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:19.876877 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 03:00:19.877042 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 03:00:19.881420 2348 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 03:00:19.888176 2348 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 03:00:19.922380 2824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:19.900271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47706 (local address 127.18.196.194:37227)
0925 03:00:19.912776 (+ 12505us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:19.912787 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:19.912820 (+ 33us) server_negotiation.cc:407] Connection header received
0925 03:00:19.912932 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:19.912943 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:19.913214 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:19.913409 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:00:19.914504 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.916251 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:19.919531 (+ 3280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.921003 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:19.921195 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:19.921386 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:19.921863 (+ 477us) server_negotiation.cc:299] Negotiation successful
0925 03:00:19.922036 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12312,"thread_start_us":12317,"threads_started":1}
I20260925 03:00:19.922370 2823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:19.899994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:37227 (local address 127.0.0.1:47706)
0925 03:00:19.911306 (+ 11312us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:19.911364 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 03:00:19.911573 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:00:19.913720 (+ 2147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:00:19.913731 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:00:19.913769 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:00:19.914261 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:19.914276 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.916419 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:19.916425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:19.919289 (+ 2864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:19.919307 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.921233 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:19.921240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:19.921436 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:19.921505 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 03:00:19.921666 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 03:00:19.921867 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10165,"mutex_wait_us":87,"thread_start_us":11871,"threads_started":1}
I20260925 03:00:19.922891 2821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:19.900090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52410 (local address 127.18.196.195:37589)
0925 03:00:19.913272 (+ 13182us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:19.913280 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:19.913315 (+ 35us) server_negotiation.cc:407] Connection header received
0925 03:00:19.913406 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:19.913413 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:19.913624 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:19.913812 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:00:19.914824 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.916415 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:19.920113 (+ 3698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.921378 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:19.921560 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:19.921743 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:19.922388 (+ 645us) server_negotiation.cc:299] Negotiation successful
0925 03:00:19.922563 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13009,"thread_start_us":12850,"threads_started":1}
I20260925 03:00:19.922947 2822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:19.900000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:37589 (local address 127.0.0.1:52410)
0925 03:00:19.912147 (+ 12147us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:19.912164 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 03:00:19.912270 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:00:19.914091 (+ 1821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:00:19.914099 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:00:19.914129 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:00:19.914596 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:19.914610 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.916666 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:19.916677 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:19.919922 (+ 3245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:19.919938 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.921582 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:19.921590 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:19.921800 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:19.921831 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 03:00:19.921913 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 03:00:19.922040 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12024,"mutex_wait_us":67,"thread_start_us":10339,"threads_started":1}
I20260925 03:00:19.926656 2610 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8b22c4d067454aba55abf89a62bddf (DEFAULT_TABLE table=test-table [id=865de962a2b644c69659c3eebb8cffad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 03:00:19.926945 2743 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8b22c4d067454aba55abf89a62bddf (DEFAULT_TABLE table=test-table [id=865de962a2b644c69659c3eebb8cffad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 03:00:19.928104 2826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:19.913376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33491 (local address 127.0.0.1:55796)
0925 03:00:19.919282 (+ 5906us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:19.919301 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 03:00:19.919434 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:00:19.920705 (+ 1271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:00:19.920710 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:00:19.920738 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:00:19.921089 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:19.921101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.923067 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:19.923075 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:19.925101 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:19.925117 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.927260 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:19.927266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:19.927460 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:19.927501 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 03:00:19.927600 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 03:00:19.927770 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5735,"thread_start_us":5589,"threads_started":1}
I20260925 03:00:19.928263 2610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8b22c4d067454aba55abf89a62bddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:00:19.928418 2743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8b22c4d067454aba55abf89a62bddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:00:19.928972 2825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:19.912792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55796 (local address 127.18.196.193:33491)
0925 03:00:19.916721 (+ 3929us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:19.916730 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:19.919431 (+ 2701us) server_negotiation.cc:407] Connection header received
0925 03:00:19.920205 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:19.920214 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:19.920457 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:19.920662 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:00:19.921308 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.922893 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:19.925290 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.926785 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:19.927077 (+ 292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:19.927247 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:19.928466 (+ 1219us) server_negotiation.cc:299] Negotiation successful
0925 03:00:19.928648 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3750,"thread_start_us":3566,"threads_started":1}
I20260925 03:00:19.932847 2474 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8b22c4d067454aba55abf89a62bddf (DEFAULT_TABLE table=test-table [id=865de962a2b644c69659c3eebb8cffad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 03:00:19.934334 2474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8b22c4d067454aba55abf89a62bddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:00:19.941891 2827 tablet_bootstrap.cc:492] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae: Bootstrap starting.
I20260925 03:00:19.943905 2828 tablet_bootstrap.cc:492] T 6c8b22c4d067454aba55abf89a62bddf P 3b5c6982fd6549ddb53e9a3aef215f1e: Bootstrap starting.
I20260925 03:00:19.945784 2827 tablet_bootstrap.cc:654] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae: Neither blocks nor log segments found. Creating new log.
I20260925 03:00:19.947122 2827 log.cc:824] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae: Log is configured to *not* fsync() on all Append() calls
I20260925 03:00:19.947495 2828 tablet_bootstrap.cc:654] T 6c8b22c4d067454aba55abf89a62bddf P 3b5c6982fd6549ddb53e9a3aef215f1e: Neither blocks nor log segments found. Creating new log.
I20260925 03:00:19.948801 2828 log.cc:824] T 6c8b22c4d067454aba55abf89a62bddf P 3b5c6982fd6549ddb53e9a3aef215f1e: Log is configured to *not* fsync() on all Append() calls
I20260925 03:00:19.949536 2831 tablet_bootstrap.cc:492] T 6c8b22c4d067454aba55abf89a62bddf P a339037fb3c14611b0ca6c8892c83e87: Bootstrap starting.
I20260925 03:00:19.950904 2827 tablet_bootstrap.cc:492] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae: No bootstrap required, opened a new log
I20260925 03:00:19.951252 2827 ts_tablet_manager.cc:1397] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260925 03:00:19.952461 2831 tablet_bootstrap.cc:654] T 6c8b22c4d067454aba55abf89a62bddf P a339037fb3c14611b0ca6c8892c83e87: Neither blocks nor log segments found. Creating new log.
I20260925 03:00:19.953332 2831 log.cc:824] T 6c8b22c4d067454aba55abf89a62bddf P a339037fb3c14611b0ca6c8892c83e87: Log is configured to *not* fsync() on all Append() calls
I20260925 03:00:19.956763 2828 tablet_bootstrap.cc:492] T 6c8b22c4d067454aba55abf89a62bddf P 3b5c6982fd6549ddb53e9a3aef215f1e: No bootstrap required, opened a new log
I20260925 03:00:19.956858 2831 tablet_bootstrap.cc:492] T 6c8b22c4d067454aba55abf89a62bddf P a339037fb3c14611b0ca6c8892c83e87: No bootstrap required, opened a new log
I20260925 03:00:19.957088 2828 ts_tablet_manager.cc:1397] T 6c8b22c4d067454aba55abf89a62bddf P 3b5c6982fd6549ddb53e9a3aef215f1e: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260925 03:00:19.957182 2831 ts_tablet_manager.cc:1397] T 6c8b22c4d067454aba55abf89a62bddf P a339037fb3c14611b0ca6c8892c83e87: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 03:00:19.962723 2827 raft_consensus.cc:348] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 37589 } } peers { permanent_uuid: "f7362975190748bd837bc87ec6f9e5ae" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37227 } } peers { permanent_uuid: "a339037fb3c14611b0ca6c8892c83e87" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33491 } }
I20260925 03:00:19.963209 2827 raft_consensus.cc:374] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:00:19.963346 2827 raft_consensus.cc:729] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7362975190748bd837bc87ec6f9e5ae, State: Initialized, Role: FOLLOWER
I20260925 03:00:19.964013 2827 consensus_queue.cc:260] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 37589 } } peers { permanent_uuid: "f7362975190748bd837bc87ec6f9e5ae" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37227 } } peers { permanent_uuid: "a339037fb3c14611b0ca6c8892c83e87" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33491 } }
I20260925 03:00:19.965185 2675 heartbeater.cc:503] Master 127.18.196.254:46387 was elected leader, sending a full tablet report...
I20260925 03:00:19.965169 2828 raft_consensus.cc:348] T 6c8b22c4d067454aba55abf89a62bddf P 3b5c6982fd6549ddb53e9a3aef215f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 37589 } } peers { permanent_uuid: "f7362975190748bd837bc87ec6f9e5ae" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37227 } } peers { permanent_uuid: "a339037fb3c14611b0ca6c8892c83e87" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33491 } }
I20260925 03:00:19.965560 2828 raft_consensus.cc:374] T 6c8b22c4d067454aba55abf89a62bddf P 3b5c6982fd6549ddb53e9a3aef215f1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:00:19.965678 2828 raft_consensus.cc:729] T 6c8b22c4d067454aba55abf89a62bddf P 3b5c6982fd6549ddb53e9a3aef215f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5c6982fd6549ddb53e9a3aef215f1e, State: Initialized, Role: FOLLOWER
I20260925 03:00:19.965848 2827 ts_tablet_manager.cc:1428] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260925 03:00:19.966318 2828 consensus_queue.cc:260] T 6c8b22c4d067454aba55abf89a62bddf P 3b5c6982fd6549ddb53e9a3aef215f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 37589 } } peers { permanent_uuid: "f7362975190748bd837bc87ec6f9e5ae" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37227 } } peers { permanent_uuid: "a339037fb3c14611b0ca6c8892c83e87" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33491 } }
I20260925 03:00:19.968672 2809 heartbeater.cc:503] Master 127.18.196.254:46387 was elected leader, sending a full tablet report...
I20260925 03:00:19.968866 2831 raft_consensus.cc:348] T 6c8b22c4d067454aba55abf89a62bddf P a339037fb3c14611b0ca6c8892c83e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 37589 } } peers { permanent_uuid: "f7362975190748bd837bc87ec6f9e5ae" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37227 } } peers { permanent_uuid: "a339037fb3c14611b0ca6c8892c83e87" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33491 } }
I20260925 03:00:19.969260 2831 raft_consensus.cc:374] T 6c8b22c4d067454aba55abf89a62bddf P a339037fb3c14611b0ca6c8892c83e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:00:19.969352 2831 raft_consensus.cc:729] T 6c8b22c4d067454aba55abf89a62bddf P a339037fb3c14611b0ca6c8892c83e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a339037fb3c14611b0ca6c8892c83e87, State: Initialized, Role: FOLLOWER
I20260925 03:00:19.969452 2828 ts_tablet_manager.cc:1428] T 6c8b22c4d067454aba55abf89a62bddf P 3b5c6982fd6549ddb53e9a3aef215f1e: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260925 03:00:19.970013 2831 consensus_queue.cc:260] T 6c8b22c4d067454aba55abf89a62bddf P a339037fb3c14611b0ca6c8892c83e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 37589 } } peers { permanent_uuid: "f7362975190748bd837bc87ec6f9e5ae" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37227 } } peers { permanent_uuid: "a339037fb3c14611b0ca6c8892c83e87" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33491 } }
I20260925 03:00:19.971788 2831 ts_tablet_manager.cc:1428] T 6c8b22c4d067454aba55abf89a62bddf P a339037fb3c14611b0ca6c8892c83e87: Time spent starting tablet: real 0.015s user 0.008s sys 0.005s
I20260925 03:00:19.992525 2834 raft_consensus.cc:482] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:00:19.992792 2834 raft_consensus.cc:504] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 37589 } } peers { permanent_uuid: "f7362975190748bd837bc87ec6f9e5ae" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37227 } } peers { permanent_uuid: "a339037fb3c14611b0ca6c8892c83e87" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33491 } }
I20260925 03:00:19.994304 2834 leader_election.cc:288] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5c6982fd6549ddb53e9a3aef215f1e (127.18.196.195:37589), a339037fb3c14611b0ca6c8892c83e87 (127.18.196.193:33491)
I20260925 03:00:20.002573 2825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:19.994853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:47801 (local address 127.18.196.193:33491)
0925 03:00:19.995064 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:19.995070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:19.995965 (+ 895us) server_negotiation.cc:407] Connection header received
0925 03:00:19.996164 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:19.996168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:19.996241 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:19.996342 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:00:19.997194 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.998331 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:20.000793 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:20.001576 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:20.001782 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:20.001867 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:20.002188 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 03:00:20.002329 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 03:00:20.002630 2821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:19.994825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:35691 (local address 127.18.196.195:37589)
0925 03:00:19.995064 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:19.995070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:19.995965 (+ 895us) server_negotiation.cc:407] Connection header received
0925 03:00:19.996150 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:19.996162 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:19.996241 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:19.996342 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:00:19.997194 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.998331 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:20.000871 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:20.001636 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:20.001839 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:20.001914 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:20.002270 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 03:00:20.002408 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260925 03:00:20.002681 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:19.994707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33491 (local address 127.18.196.194:47801)
0925 03:00:19.995836 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:19.995853 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 03:00:19.995997 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:00:19.996381 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:00:19.996386 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:00:19.996414 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:00:19.997006 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:19.997020 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.998482 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:19.998487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:20.000626 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:20.000640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:20.001830 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:20.001837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:20.001994 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:20.002018 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 03:00:20.002128 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 03:00:20.002262 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"mutex_wait_us":124,"thread_start_us":699,"threads_started":1}
I20260925 03:00:20.003445 2837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:19.994579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:37589 (local address 127.18.196.194:35691)
0925 03:00:19.995802 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:19.995827 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 03:00:19.996014 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:00:19.996383 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:00:19.996387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:00:19.996425 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:00:19.997013 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:19.997022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:19.998488 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:19.998492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:20.000686 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:20.000701 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:20.001867 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:20.001873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:20.002072 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:20.002116 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 03:00:20.002894 (+ 778us) client_negotiation.cc:240] Negotiation successful
0925 03:00:20.003022 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1057,"mutex_wait_us":209,"spinlock_wait_cycles":70784,"thread_start_us":801,"threads_started":1}
I20260925 03:00:20.004024 2495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8b22c4d067454aba55abf89a62bddf" candidate_uuid: "f7362975190748bd837bc87ec6f9e5ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a339037fb3c14611b0ca6c8892c83e87" is_pre_election: true
I20260925 03:00:20.004436 2764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8b22c4d067454aba55abf89a62bddf" candidate_uuid: "f7362975190748bd837bc87ec6f9e5ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e" is_pre_election: true
I20260925 03:00:20.004658 2495 raft_consensus.cc:2454] T 6c8b22c4d067454aba55abf89a62bddf P a339037fb3c14611b0ca6c8892c83e87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7362975190748bd837bc87ec6f9e5ae in term 0.
I20260925 03:00:20.005012 2764 raft_consensus.cc:2454] T 6c8b22c4d067454aba55abf89a62bddf P 3b5c6982fd6549ddb53e9a3aef215f1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7362975190748bd837bc87ec6f9e5ae in term 0.
I20260925 03:00:20.005376 2565 leader_election.cc:302] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a339037fb3c14611b0ca6c8892c83e87, f7362975190748bd837bc87ec6f9e5ae; no voters:
I20260925 03:00:20.005820 2834 raft_consensus.cc:2781] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:00:20.005980 2834 raft_consensus.cc:482] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:00:20.006099 2834 raft_consensus.cc:3037] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:00:20.008404 2834 raft_consensus.cc:504] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 37589 } } peers { permanent_uuid: "f7362975190748bd837bc87ec6f9e5ae" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37227 } } peers { permanent_uuid: "a339037fb3c14611b0ca6c8892c83e87" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33491 } }
I20260925 03:00:20.009039 2834 leader_election.cc:288] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae [CANDIDATE]: Term 1 election: Requested vote from peers 3b5c6982fd6549ddb53e9a3aef215f1e (127.18.196.195:37589), a339037fb3c14611b0ca6c8892c83e87 (127.18.196.193:33491)
I20260925 03:00:20.009588 2495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8b22c4d067454aba55abf89a62bddf" candidate_uuid: "f7362975190748bd837bc87ec6f9e5ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a339037fb3c14611b0ca6c8892c83e87"
I20260925 03:00:20.009588 2764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8b22c4d067454aba55abf89a62bddf" candidate_uuid: "f7362975190748bd837bc87ec6f9e5ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e"
I20260925 03:00:20.009809 2764 raft_consensus.cc:3037] T 6c8b22c4d067454aba55abf89a62bddf P 3b5c6982fd6549ddb53e9a3aef215f1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:00:20.009809 2495 raft_consensus.cc:3037] T 6c8b22c4d067454aba55abf89a62bddf P a339037fb3c14611b0ca6c8892c83e87 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:00:20.013104 2764 raft_consensus.cc:2454] T 6c8b22c4d067454aba55abf89a62bddf P 3b5c6982fd6549ddb53e9a3aef215f1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7362975190748bd837bc87ec6f9e5ae in term 1.
I20260925 03:00:20.013105 2495 raft_consensus.cc:2454] T 6c8b22c4d067454aba55abf89a62bddf P a339037fb3c14611b0ca6c8892c83e87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7362975190748bd837bc87ec6f9e5ae in term 1.
I20260925 03:00:20.013585 2565 leader_election.cc:302] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3b5c6982fd6549ddb53e9a3aef215f1e, a339037fb3c14611b0ca6c8892c83e87, f7362975190748bd837bc87ec6f9e5ae; no voters:
I20260925 03:00:20.013991 2834 raft_consensus.cc:2781] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:00:20.014477 2834 raft_consensus.cc:686] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae [term 1 LEADER]: Becoming Leader. State: Replica: f7362975190748bd837bc87ec6f9e5ae, State: Running, Role: LEADER
I20260925 03:00:20.015040 2834 consensus_queue.cc:237] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 37589 } } peers { permanent_uuid: "f7362975190748bd837bc87ec6f9e5ae" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37227 } } peers { permanent_uuid: "a339037fb3c14611b0ca6c8892c83e87" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33491 } }
I20260925 03:00:20.019886 2347 catalog_manager.cc:5495] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae reported cstate change: term changed from 0 to 1, leader changed from <none> to f7362975190748bd837bc87ec6f9e5ae (127.18.196.194). New cstate: current_term: 1 leader_uuid: "f7362975190748bd837bc87ec6f9e5ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 37589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7362975190748bd837bc87ec6f9e5ae" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a339037fb3c14611b0ca6c8892c83e87" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33491 } health_report { overall_health: UNKNOWN } } }
I20260925 03:00:20.040578 2824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:20.034882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47720 (local address 127.18.196.194:37227)
0925 03:00:20.035088 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:20.035096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:20.035155 (+ 59us) server_negotiation.cc:407] Connection header received
0925 03:00:20.035483 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:20.035487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:20.035573 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:20.035748 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:00:20.036124 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:20.036857 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:20.038369 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:20.038765 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:20.038931 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:20.039457 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:00:20.039972 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:00:20.040130 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:20.040217 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 03:00:20.040298 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 03:00:20.048729 2495 raft_consensus.cc:1261] T 6c8b22c4d067454aba55abf89a62bddf P a339037fb3c14611b0ca6c8892c83e87 [term 1 FOLLOWER]: Refusing update from remote peer f7362975190748bd837bc87ec6f9e5ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:00:20.048710 2764 raft_consensus.cc:1261] T 6c8b22c4d067454aba55abf89a62bddf P 3b5c6982fd6549ddb53e9a3aef215f1e [term 1 FOLLOWER]: Refusing update from remote peer f7362975190748bd837bc87ec6f9e5ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:00:20.049990 2839 consensus_queue.cc:1035] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a339037fb3c14611b0ca6c8892c83e87" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:00:20.050377 2834 consensus_queue.cc:1035] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 37589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:00:20.063589 2842 mvcc.cc:205] Tried to move back new op lower bound from 7333090181308907520 to 7333090181185683456. Current Snapshot: MvccSnapshot[applied={T|T < 7333090181308907520}]
I20260925 03:00:20.079639 2347 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 03:00:20.079952 2347 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 03:00:20.083648 2347 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb: Sending DeleteTablet for 3 replicas of tablet 6c8b22c4d067454aba55abf89a62bddf
I20260925 03:00:20.084779 2743 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8b22c4d067454aba55abf89a62bddf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 03:00:20 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:52410
I20260925 03:00:20.085086 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 2410
I20260925 03:00:20.085283 2610 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8b22c4d067454aba55abf89a62bddf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 03:00:20 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:47706
I20260925 03:00:20.086016 2474 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8b22c4d067454aba55abf89a62bddf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 03:00:20 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:55796
I20260925 03:00:20.097452 2852 tablet_replica.cc:330] T 6c8b22c4d067454aba55abf89a62bddf P 3b5c6982fd6549ddb53e9a3aef215f1e: stopping tablet replica
I20260925 03:00:20.098054 2855 tablet_replica.cc:330] T 6c8b22c4d067454aba55abf89a62bddf P a339037fb3c14611b0ca6c8892c83e87: stopping tablet replica
I20260925 03:00:20.098703 2855 raft_consensus.cc:2229] T 6c8b22c4d067454aba55abf89a62bddf P a339037fb3c14611b0ca6c8892c83e87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:00:20.099255 2852 raft_consensus.cc:2229] T 6c8b22c4d067454aba55abf89a62bddf P 3b5c6982fd6549ddb53e9a3aef215f1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:00:20.099884 2855 raft_consensus.cc:2258] T 6c8b22c4d067454aba55abf89a62bddf P a339037fb3c14611b0ca6c8892c83e87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:00:20.100404 2852 raft_consensus.cc:2258] T 6c8b22c4d067454aba55abf89a62bddf P 3b5c6982fd6549ddb53e9a3aef215f1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:00:20.101058 2853 tablet_replica.cc:330] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae: stopping tablet replica
I20260925 03:00:20.101543 2853 raft_consensus.cc:2229] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae [term 1 LEADER]: Raft consensus shutting down.
I20260925 03:00:20.102288 2853 raft_consensus.cc:2258] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:00:20.103698 2852 ts_tablet_manager.cc:1905] T 6c8b22c4d067454aba55abf89a62bddf P 3b5c6982fd6549ddb53e9a3aef215f1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 03:00:20.104225 2855 ts_tablet_manager.cc:1905] T 6c8b22c4d067454aba55abf89a62bddf P a339037fb3c14611b0ca6c8892c83e87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 03:00:20.105830 2853 ts_tablet_manager.cc:1905] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 03:00:20.110931 2852 ts_tablet_manager.cc:1918] T 6c8b22c4d067454aba55abf89a62bddf P 3b5c6982fd6549ddb53e9a3aef215f1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 03:00:20.111189 2852 log.cc:1196] T 6c8b22c4d067454aba55abf89a62bddf P 3b5c6982fd6549ddb53e9a3aef215f1e: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/6c8b22c4d067454aba55abf89a62bddf
I20260925 03:00:20.111349 2855 ts_tablet_manager.cc:1918] T 6c8b22c4d067454aba55abf89a62bddf P a339037fb3c14611b0ca6c8892c83e87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 03:00:20.111608 2855 log.cc:1196] T 6c8b22c4d067454aba55abf89a62bddf P a339037fb3c14611b0ca6c8892c83e87: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/wal/wals/6c8b22c4d067454aba55abf89a62bddf
I20260925 03:00:20.111822 2852 ts_tablet_manager.cc:1939] T 6c8b22c4d067454aba55abf89a62bddf P 3b5c6982fd6549ddb53e9a3aef215f1e: Deleting consensus metadata
I20260925 03:00:20.112208 2855 ts_tablet_manager.cc:1939] T 6c8b22c4d067454aba55abf89a62bddf P a339037fb3c14611b0ca6c8892c83e87: Deleting consensus metadata
I20260925 03:00:20.113440 2853 ts_tablet_manager.cc:1918] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 03:00:20.113945 2332 catalog_manager.cc:4841] TS 3b5c6982fd6549ddb53e9a3aef215f1e (127.18.196.195:37589): tablet 6c8b22c4d067454aba55abf89a62bddf (table test-table [id=865de962a2b644c69659c3eebb8cffad]) successfully deleted
I20260925 03:00:20.114151 2853 log.cc:1196] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/6c8b22c4d067454aba55abf89a62bddf
I20260925 03:00:20.114353 2334 catalog_manager.cc:4841] TS a339037fb3c14611b0ca6c8892c83e87 (127.18.196.193:33491): tablet 6c8b22c4d067454aba55abf89a62bddf (table test-table [id=865de962a2b644c69659c3eebb8cffad]) successfully deleted
I20260925 03:00:20.114740 2853 ts_tablet_manager.cc:1939] T 6c8b22c4d067454aba55abf89a62bddf P f7362975190748bd837bc87ec6f9e5ae: Deleting consensus metadata
I20260925 03:00:20.116508 2334 catalog_manager.cc:4841] TS f7362975190748bd837bc87ec6f9e5ae (127.18.196.194:37227): tablet 6c8b22c4d067454aba55abf89a62bddf (table test-table [id=865de962a2b644c69659c3eebb8cffad]) successfully deleted
Sep 25 03:00:20 dist-test-slave-8zvk krb5kdc[2303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.193@KRBTEST.COM
I20260925 03:00:20.127134 2825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:20.086243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55804 (local address 127.18.196.193:33491)
0925 03:00:20.086423 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:20.086430 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:20.097033 (+ 10603us) server_negotiation.cc:407] Connection header received
0925 03:00:20.097588 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:20.097599 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:20.097733 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:20.097841 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:00:20.101302 (+ 3461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:20.102478 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:20.113695 (+ 11217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:20.114375 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:20.114520 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:20.119903 (+ 5383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:00:20.119922 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:00:20.119942 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:00:20.119991 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:00:20.124043 (+ 4052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:20.124792 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:20.124804 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:20.124808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:20.124999 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:20.125385 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:20.125391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:20.125395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:20.125899 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:00:20.126222 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:20.126446 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 03:00:20.126683 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 03:00:20.127892 2535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:00:20.216804 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 2410
I20260925 03:00:20.234943 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 2547
Sep 25 03:00:20 dist-test-slave-8zvk krb5kdc[2303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.194@KRBTEST.COM
I20260925 03:00:20.252899 2824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:20.235842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.18.196.194:37227)
0925 03:00:20.236041 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:20.236048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:20.236070 (+ 22us) server_negotiation.cc:407] Connection header received
0925 03:00:20.236140 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:20.236146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:20.236228 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:20.236344 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:00:20.237575 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:20.238772 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:20.240543 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:20.241180 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:20.241291 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:20.246244 (+ 4953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:00:20.246259 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:00:20.246278 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:00:20.246322 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:00:20.249984 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:20.250662 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:20.250672 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:20.250678 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:20.250873 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:20.251280 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:20.251285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:20.251289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:20.251724 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:00:20.252055 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:20.252295 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 03:00:20.252507 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 03:00:20.253643 2670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:00:20.363145 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 2547
I20260925 03:00:20.382757 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 2680
Sep 25 03:00:20 dist-test-slave-8zvk krb5kdc[2303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.195@KRBTEST.COM
I20260925 03:00:20.397469 2821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:20.383487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52414 (local address 127.18.196.195:37589)
0925 03:00:20.383658 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:20.383663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:20.383786 (+ 123us) server_negotiation.cc:407] Connection header received
0925 03:00:20.383915 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:20.383919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:20.383982 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:20.384064 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:00:20.385265 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:20.386071 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:20.387579 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:20.387953 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:20.388042 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:20.392299 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:00:20.392311 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:00:20.392325 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:00:20.392359 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:00:20.395152 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:20.395853 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:20.395860 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:20.395862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:20.395988 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:20.396297 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:20.396300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:20.396302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:20.396626 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:00:20.396862 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:20.397078 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 03:00:20.397224 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 03:00:20.398099 2804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:00:20.487426 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 2680
I20260925 03:00:20.504367 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 2316
I20260925 03:00:20.505571 2378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:00:20.614675 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 2316
I20260925 03:00:40.636582 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:46387
--webserver_interface=127.18.196.254
--webserver_port=35321
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:46387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:00:40.945644 2860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:00:40.945924 2860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:00:40.946098 2860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:00:40.953109 2860 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 03:00:40.953195 2860 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 03:00:40.953287 2860 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 03:00:40.953361 2860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:00:40.953454 2860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:00:40.953552 2860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:00:40.953616 2860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:00:40.953687 2860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:00:40.963114 2860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:46387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:46387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=35321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:00:40.964314 2860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:00:40.965557 2860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 03:00:40.987547 2860 server_base.cc:1034] running on GCE node
W20260925 03:00:40.987560 2867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:00:40.987771 2866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:00:40.987524 2869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:00:40.988912 2860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:00:40.990332 2860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:00:40.991581 2860 hybrid_clock.cc:648] HybridClock initialized: now 1790305240991523 us; error 48 us; skew 500 ppm
I20260925 03:00:40.991972 2860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:00:40 dist-test-slave-8zvk krb5kdc[2303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305240, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:00:41.000115 2860 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 03:00:41.001626 2860 webserver.cc:460] Webserver started at http://127.18.196.254:35321/ using document root <none> and password file <none>
I20260925 03:00:41.002336 2860 fs_manager.cc:359] Metadata directory not provided
I20260925 03:00:41.002437 2860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:00:41.008038 2860 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 03:00:41.010942 2876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:41.012243 2860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 03:00:41.012434 2860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "89ae4c9f9042478499d8c6a40f3cc0bb"
format_stamp: "Formatted at 2026-09-25 03:00:18 on dist-test-slave-8zvk"
server_key: "880f2ee6192cb43171778870400ebfd1"
server_key_iv: "ceb103f29afb2173dde1e580916dca46"
server_key_version: "encryptionkey@0"
I20260925 03:00:41.013450 2860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:00:41.031656 2860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:00:41.035458 2860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:00:41.035751 2860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:00:41.054104 2860 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:46387
I20260925 03:00:41.054141 2928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:46387 every 8 connection(s)
I20260925 03:00:41.055588 2860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:00:41.059803 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 2860
I20260925 03:00:41.060839 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:33491
--local_ip_for_outbound_sockets=127.18.196.193
--tserver_master_addrs=127.18.196.254:46387
--webserver_port=43231
--webserver_interface=127.18.196.193
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 03:00:41.063596 2929 sys_catalog.cc:263] Verifying existing consensus state
I20260925 03:00:41.068756 2929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb: Bootstrap starting.
I20260925 03:00:41.088928 2929 log.cc:824] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb: Log is configured to *not* fsync() on all Append() calls
I20260925 03:00:41.101233 2929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 03:00:41.101795 2929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb: Bootstrap complete.
I20260925 03:00:41.110718 2929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae4c9f9042478499d8c6a40f3cc0bb" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 46387 } }
I20260925 03:00:41.111647 2929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89ae4c9f9042478499d8c6a40f3cc0bb, State: Initialized, Role: FOLLOWER
I20260925 03:00:41.112282 2929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae4c9f9042478499d8c6a40f3cc0bb" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 46387 } }
I20260925 03:00:41.112528 2929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:00:41.112670 2929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:00:41.112861 2929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 1 FOLLOWER]: Advancing to term 2
I20260925 03:00:41.116719 2929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae4c9f9042478499d8c6a40f3cc0bb" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 46387 } }
I20260925 03:00:41.117295 2929 leader_election.cc:302] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89ae4c9f9042478499d8c6a40f3cc0bb; no voters:
I20260925 03:00:41.117959 2929 leader_election.cc:288] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 03:00:41.118124 2933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 2 FOLLOWER]: Leader election won for term 2
I20260925 03:00:41.120440 2929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:00:41.123585 2933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 2 LEADER]: Becoming Leader. State: Replica: 89ae4c9f9042478499d8c6a40f3cc0bb, State: Running, Role: LEADER
I20260925 03:00:41.124352 2933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae4c9f9042478499d8c6a40f3cc0bb" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 46387 } }
I20260925 03:00:41.127542 2935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89ae4c9f9042478499d8c6a40f3cc0bb. Latest consensus state: current_term: 2 leader_uuid: "89ae4c9f9042478499d8c6a40f3cc0bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae4c9f9042478499d8c6a40f3cc0bb" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 46387 } } }
I20260925 03:00:41.133018 2935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [sys.catalog]: This master's current role is: LEADER
I20260925 03:00:41.127172 2934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "89ae4c9f9042478499d8c6a40f3cc0bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae4c9f9042478499d8c6a40f3cc0bb" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 46387 } } }
I20260925 03:00:41.133579 2934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [sys.catalog]: This master's current role is: LEADER
I20260925 03:00:41.141227 2943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:00:41.149384 2943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:00:41.150920 2943 catalog_manager.cc:1259] Loaded cluster ID: fb3e3f55110242f18de7b693acb25141
I20260925 03:00:41.151081 2943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:00:41.155220 2943 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:00:41.156818 2943 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb: Loaded TSK: 0
I20260925 03:00:41.160882 2950 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 03:00:41.189254 2943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb: Generated new TSK 1
I20260925 03:00:41.189648 2943 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 03:00:41.409097 2931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:00:41.409356 2931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:00:41.409503 2931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:00:41.416746 2931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:00:41.416848 2931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:00:41.417048 2931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 03:00:41.427078 2931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:33491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=43231
--tserver_master_addrs=127.18.196.254:46387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:00:41.428402 2931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:00:41.429726 2931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:00:41.443850 2956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:00:41.443872 2957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:00:41.444401 2959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:00:41.445629 2931 server_base.cc:1034] running on GCE node
I20260925 03:00:41.446439 2931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:00:41.447728 2931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:00:41.448994 2931 hybrid_clock.cc:648] HybridClock initialized: now 1790305241448921 us; error 59 us; skew 500 ppm
I20260925 03:00:41.449400 2931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:00:41 dist-test-slave-8zvk krb5kdc[2303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305241, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:00:41.453379 2931 init.cc:377] Logged in from keytab as kudu/127.18.196.193@KRBTEST.COM (short username kudu)
I20260925 03:00:41.454984 2931 webserver.cc:460] Webserver started at http://127.18.196.193:43231/ using document root <none> and password file <none>
I20260925 03:00:41.455704 2931 fs_manager.cc:359] Metadata directory not provided
I20260925 03:00:41.455850 2931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:00:41.461410 2931 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 03:00:41.464447 2966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:41.465879 2931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 03:00:41.466045 2931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "a339037fb3c14611b0ca6c8892c83e87"
format_stamp: "Formatted at 2026-09-25 03:00:18 on dist-test-slave-8zvk"
server_key: "aefd6e2d7f6c33e9f1bd04ffad4905c4"
server_key_iv: "bab59001ce0220117fe1ed39b6094f19"
server_key_version: "encryptionkey@0"
I20260925 03:00:41.467051 2931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:00:41.481489 2931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:00:41.485803 2931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:00:41.486078 2931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:00:41.487087 2931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:00:41.488448 2931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:00:41.488572 2931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:41.488693 2931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:00:41.488782 2931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:41.526500 2931 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:33491
I20260925 03:00:41.526584 3079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:33491 every 8 connection(s)
I20260925 03:00:41.527935 2931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 03:00:41.538338 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 2931
I20260925 03:00:41.539124 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:37227
--local_ip_for_outbound_sockets=127.18.196.194
--tserver_master_addrs=127.18.196.254:46387
--webserver_port=33743
--webserver_interface=127.18.196.194
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 03:00:41 dist-test-slave-8zvk krb5kdc[2303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305241, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:00:41.550829 3083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:41.530939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.193:38637)
0925 03:00:41.531791 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:41.531837 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 03:00:41.533522 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:00:41.534744 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:00:41.534751 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:00:41.535331 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:00:41.535835 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:41.535853 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:41.537409 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:41.537414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:41.538676 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:41.538688 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:41.540183 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:41.540189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:41.540379 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:41.541039 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:00:41.541065 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:00:41.543709 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:00:41.546985 (+ 3276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:00:41.546996 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:00:41.547013 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:00:41.547444 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:00:41.547951 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:00:41.547954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:00:41.547956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:00:41.548122 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:00:41.549027 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:00:41.549033 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:00:41.549305 (+ 272us) client_negotiation.cc:769] Sending connection context
0925 03:00:41.549514 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 03:00:41.549881 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":359,"threads_started":1}
I20260925 03:00:41.551409 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:41.530616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:38637 (local address 127.18.196.254:46387)
0925 03:00:41.531261 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:41.531272 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:41.531989 (+ 717us) server_negotiation.cc:407] Connection header received
0925 03:00:41.534004 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:41.534017 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:41.534327 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:41.534616 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:00:41.536002 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:41.537173 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:41.538822 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:41.539971 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:41.540166 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:41.543977 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:00:41.543984 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:00:41.543996 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:00:41.544031 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:00:41.546800 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:41.547591 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:41.547601 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:41.547610 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:41.547837 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:41.548244 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:41.548252 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:41.548255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:41.548898 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:00:41.549088 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:41.549778 (+ 690us) server_negotiation.cc:299] Negotiation successful
0925 03:00:41.550199 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":279,"threads_started":1}
I20260925 03:00:41.554040 3080 heartbeater.cc:347] Connected to a master server at 127.18.196.254:46387
I20260925 03:00:41.554344 3080 heartbeater.cc:464] Registering TS with master...
I20260925 03:00:41.555125 3080 heartbeater.cc:511] Master 127.18.196.254:46387 requested a full tablet report, sending...
I20260925 03:00:41.557220 2893 ts_manager.cc:194] Registered new tserver with Master: a339037fb3c14611b0ca6c8892c83e87 (127.18.196.193:33491)
I20260925 03:00:41.558696 2893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.193@KRBTEST.COM'} at 127.18.196.193:38637
W20260925 03:00:41.865381 3084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:00:41.865658 3084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:00:41.865801 3084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:00:41.873219 3084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:00:41.873330 3084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:00:41.873530 3084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 03:00:41.883128 3084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:37227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=33743
--tserver_master_addrs=127.18.196.254:46387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:00:41.884387 3084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:00:41.885742 3084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:00:41.898443 3092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:00:41.898478 3089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:00:41.898453 3090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:00:41.899307 3084 server_base.cc:1034] running on GCE node
I20260925 03:00:41.899904 3084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:00:41.900904 3084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:00:41.902123 3084 hybrid_clock.cc:648] HybridClock initialized: now 1790305241902085 us; error 52 us; skew 500 ppm
I20260925 03:00:41.902449 3084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:00:41 dist-test-slave-8zvk krb5kdc[2303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305241, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:00:41.905563 3084 init.cc:377] Logged in from keytab as kudu/127.18.196.194@KRBTEST.COM (short username kudu)
I20260925 03:00:41.907178 3084 webserver.cc:460] Webserver started at http://127.18.196.194:33743/ using document root <none> and password file <none>
I20260925 03:00:41.907816 3084 fs_manager.cc:359] Metadata directory not provided
I20260925 03:00:41.907927 3084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:00:41.913103 3084 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 03:00:41.915879 3099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:41.917243 3084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 03:00:41.917372 3084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "f7362975190748bd837bc87ec6f9e5ae"
format_stamp: "Formatted at 2026-09-25 03:00:19 on dist-test-slave-8zvk"
server_key: "623b0f2782b866c6d9a49f0a302d0c57"
server_key_iv: "c9826eebbf8b47ede715b069130ec636"
server_key_version: "encryptionkey@0"
I20260925 03:00:41.918356 3084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:00:41.935757 3084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:00:41.939634 3084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:00:41.939855 3084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:00:41.940737 3084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:00:41.941987 3084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:00:41.942070 3084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:41.942140 3084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:00:41.942198 3084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:41.975462 3084 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:37227
I20260925 03:00:41.975545 3212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:37227 every 8 connection(s)
I20260925 03:00:41.976881 3084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 03:00:41.986778 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 3084
I20260925 03:00:41.987502 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:37589
--local_ip_for_outbound_sockets=127.18.196.195
--tserver_master_addrs=127.18.196.254:46387
--webserver_port=40497
--webserver_interface=127.18.196.195
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 03:00:41 dist-test-slave-8zvk krb5kdc[2303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305241, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:00:41.995759 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:41.979326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:33811 (local address 127.18.196.254:46387)
0925 03:00:41.979524 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:41.979529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:41.980571 (+ 1042us) server_negotiation.cc:407] Connection header received
0925 03:00:41.982393 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:41.982399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:41.982492 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:41.982601 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:00:41.984074 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:41.984750 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:41.986266 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:41.986677 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:41.986806 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:41.990388 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:00:41.990393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:00:41.990396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:00:41.990426 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:00:41.992776 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:41.993494 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:41.993499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:41.993503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:41.993625 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:41.994051 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:41.994056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:41.994060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:41.994332 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:00:41.994491 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:41.995193 (+ 702us) server_negotiation.cc:299] Negotiation successful
0925 03:00:41.995418 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 03:00:41.996685 3215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:41.979743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.194:33811)
0925 03:00:41.980419 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:41.980455 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 03:00:41.982119 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:00:41.982871 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:00:41.982885 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:00:41.983366 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:00:41.983905 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:41.983923 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:41.984911 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:41.984919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:41.986113 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:41.986125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:41.986925 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:41.986933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:41.987175 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:41.987892 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:00:41.987914 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:00:41.990161 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:00:41.992949 (+ 2788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:00:41.992957 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:00:41.992971 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:00:41.993361 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:00:41.993734 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:00:41.993739 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:00:41.993743 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:00:41.993930 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:00:41.994461 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:00:41.994469 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:00:41.994783 (+ 314us) client_negotiation.cc:769] Sending connection context
0925 03:00:41.995096 (+ 313us) client_negotiation.cc:240] Negotiation successful
0925 03:00:41.995604 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":262,"threads_started":1}
I20260925 03:00:41.998122 3213 heartbeater.cc:347] Connected to a master server at 127.18.196.254:46387
I20260925 03:00:41.998427 3213 heartbeater.cc:464] Registering TS with master...
I20260925 03:00:41.999300 3213 heartbeater.cc:511] Master 127.18.196.254:46387 requested a full tablet report, sending...
I20260925 03:00:42.000835 2893 ts_manager.cc:194] Registered new tserver with Master: f7362975190748bd837bc87ec6f9e5ae (127.18.196.194:37227)
I20260925 03:00:42.002195 2893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.194@KRBTEST.COM'} at 127.18.196.194:33811
W20260925 03:00:42.316432 3216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:00:42.316673 3216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:00:42.316787 3216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:00:42.323968 3216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:00:42.324085 3216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:00:42.324254 3216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 03:00:42.333598 3216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:37589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=40497
--tserver_master_addrs=127.18.196.254:46387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:00:42.334697 3216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:00:42.335942 3216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:00:42.348405 3221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:00:42.348457 3224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:00:42.348474 3222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:00:42.349437 3216 server_base.cc:1034] running on GCE node
I20260925 03:00:42.349956 3216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:00:42.350813 3216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:00:42.352059 3216 hybrid_clock.cc:648] HybridClock initialized: now 1790305242352009 us; error 45 us; skew 500 ppm
I20260925 03:00:42.352376 3216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:00:42 dist-test-slave-8zvk krb5kdc[2303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305242, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:00:42.355285 3216 init.cc:377] Logged in from keytab as kudu/127.18.196.195@KRBTEST.COM (short username kudu)
I20260925 03:00:42.356600 3216 webserver.cc:460] Webserver started at http://127.18.196.195:40497/ using document root <none> and password file <none>
I20260925 03:00:42.357180 3216 fs_manager.cc:359] Metadata directory not provided
I20260925 03:00:42.357339 3216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:00:42.362226 3216 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260925 03:00:42.364764 3231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:42.365998 3216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 03:00:42.366192 3216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "3b5c6982fd6549ddb53e9a3aef215f1e"
format_stamp: "Formatted at 2026-09-25 03:00:19 on dist-test-slave-8zvk"
server_key: "c3d41831a73b82a666f4636dbef811bb"
server_key_iv: "6c213b760fb3411753b4f2f5602f6a17"
server_key_version: "encryptionkey@0"
I20260925 03:00:42.367173 3216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:00:42.389322 3216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:00:42.393886 3216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:00:42.394163 3216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:00:42.395314 3216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:00:42.397018 3216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:00:42.397130 3216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:42.397243 3216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:00:42.397322 3216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:00:42.437670 3216 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:37589
I20260925 03:00:42.437778 3344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:37589 every 8 connection(s)
I20260925 03:00:42.439189 3216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 03:00:42.443378 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 3216
Sep 25 03:00:42 dist-test-slave-8zvk krb5kdc[2303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305242, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:00:42.464239 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:42.441818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:35923 (local address 127.18.196.254:46387)
0925 03:00:42.442032 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:42.442038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:42.443239 (+ 1201us) server_negotiation.cc:407] Connection header received
0925 03:00:42.446272 (+ 3033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:42.446278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:42.446360 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:42.446458 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:00:42.448561 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.449868 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.452382 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.453027 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.453122 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:42.456628 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:00:42.456635 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:00:42.456644 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:00:42.456709 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:00:42.460915 (+ 4206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:42.461903 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:42.461907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:42.461909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:42.462064 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:42.462522 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:42.462526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:42.462530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:42.462912 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:00:42.463041 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:42.463828 (+ 787us) server_negotiation.cc:299] Negotiation successful
0925 03:00:42.463973 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":73,"spinlock_wait_cycles":8960}
I20260925 03:00:42.465571 3347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:42.442301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.195:35923)
0925 03:00:42.443080 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:42.443129 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 03:00:42.445920 (+ 2791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:00:42.446881 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:00:42.446893 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:00:42.447621 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:00:42.448389 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:42.448415 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.450035 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.450044 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:42.452217 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:42.452245 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.453382 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.453387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:42.453560 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:42.454163 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:00:42.454193 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:00:42.456442 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:00:42.461094 (+ 4652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:00:42.461104 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:00:42.461138 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:00:42.461782 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:00:42.462195 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:00:42.462201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:00:42.462204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:00:42.462400 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:00:42.463038 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:00:42.463048 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:00:42.463464 (+ 416us) client_negotiation.cc:769] Sending connection context
0925 03:00:42.463750 (+ 286us) client_negotiation.cc:240] Negotiation successful
0925 03:00:42.464267 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":278,"threads_started":1}
I20260925 03:00:42.466969 3345 heartbeater.cc:347] Connected to a master server at 127.18.196.254:46387
I20260925 03:00:42.467295 3345 heartbeater.cc:464] Registering TS with master...
I20260925 03:00:42.468065 3345 heartbeater.cc:511] Master 127.18.196.254:46387 requested a full tablet report, sending...
I20260925 03:00:42.469416 3349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:42.444315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32800 (local address 127.18.196.254:46387)
0925 03:00:42.446630 (+ 2315us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:42.446634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:42.446648 (+ 14us) server_negotiation.cc:407] Connection header received
0925 03:00:42.446695 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:42.446698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:42.446747 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:42.446883 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:00:42.448534 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.449868 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.452177 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.452908 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.453027 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:42.456653 (+ 3626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:00:42.456659 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:00:42.456666 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:00:42.456709 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:00:42.460975 (+ 4266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:42.462728 (+ 1753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:42.462733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:42.462736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:42.462904 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:42.466446 (+ 3542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:42.466450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:42.466452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:42.466744 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:00:42.468820 (+ 2076us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:42.468907 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 03:00:42.469121 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":168,"server-negotiator.queue_time_us":2228,"spinlock_wait_cycles":7680,"thread_start_us":2124,"threads_started":1}
I20260925 03:00:42.469574 2893 ts_manager.cc:194] Registered new tserver with Master: 3b5c6982fd6549ddb53e9a3aef215f1e (127.18.196.195:37589)
I20260925 03:00:42.470865 2893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.195@KRBTEST.COM'} at 127.18.196.195:35923
I20260925 03:00:42.473114 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 03:00:42.479899 3349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:42.474068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32816 (local address 127.18.196.254:46387)
0925 03:00:42.474233 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:42.474237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:42.474688 (+ 451us) server_negotiation.cc:407] Connection header received
0925 03:00:42.474935 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:42.474941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:42.475063 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:42.475204 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:00:42.475628 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.476315 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.477646 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.478080 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.478425 (+ 345us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:42.478992 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:00:42.479262 (+ 270us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 03:00:42.479666 (+ 404us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32816: token expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 03:00:42.480026 3349 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32816: token expired
W20260925 03:00:42.480604 2812 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.18.196.254:46387)
I20260925 03:00:42.492959 3349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:42.481162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32828 (local address 127.18.196.254:46387)
0925 03:00:42.481308 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:42.481314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:42.481385 (+ 71us) server_negotiation.cc:407] Connection header received
0925 03:00:42.481569 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:42.481573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:42.481636 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:42.481743 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:00:42.483089 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.483742 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.485030 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.485426 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.485559 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:42.488061 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:00:42.488067 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:00:42.488071 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:00:42.488107 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:00:42.490374 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:42.491056 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:42.491062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:42.491070 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:42.491214 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:42.491603 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:42.491608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:42.491611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:42.491892 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:00:42.492137 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:42.492490 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 03:00:42.492686 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 03:00:42.495994 2893 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 03:00:42.503086 2893 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 03:00:42.527519 3358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:42.514647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45796 (local address 127.18.196.195:37589)
0925 03:00:42.517401 (+ 2754us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:42.517412 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:42.517451 (+ 39us) server_negotiation.cc:407] Connection header received
0925 03:00:42.517849 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:42.517861 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:42.518178 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:42.518414 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:00:42.519896 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.521655 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.524680 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.526175 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.526333 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:42.526447 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:42.527034 (+ 587us) server_negotiation.cc:299] Negotiation successful
0925 03:00:42.527185 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2533,"thread_start_us":2329,"threads_started":1}
I20260925 03:00:42.527590 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:42.514074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:37589 (local address 127.0.0.1:45796)
0925 03:00:42.516090 (+ 2016us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:42.516115 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 03:00:42.516227 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:00:42.518453 (+ 2226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:00:42.518464 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:00:42.518507 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:00:42.519670 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:42.519681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.521843 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.521849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:42.524491 (+ 2642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:42.524507 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.526599 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.526603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:42.526739 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:42.526768 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 03:00:42.526963 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 03:00:42.527144 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1883,"mutex_wait_us":136,"thread_start_us":4895,"threads_started":1}
I20260925 03:00:42.528287 3357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:42.515519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:37227 (local address 127.0.0.1:37942)
0925 03:00:42.519369 (+ 3850us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:42.519382 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 03:00:42.519517 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:00:42.520109 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:00:42.520115 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:00:42.520149 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:00:42.520570 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:42.520581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.522503 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.522512 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:42.524854 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:42.524868 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.527468 (+ 2600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.527475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:42.527660 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:42.527694 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 03:00:42.527785 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 03:00:42.527905 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3736,"mutex_wait_us":50,"thread_start_us":778,"threads_started":1}
I20260925 03:00:42.528371 3360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:42.516076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37942 (local address 127.18.196.194:37227)
0925 03:00:42.517717 (+ 1641us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:42.517729 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:42.519477 (+ 1748us) server_negotiation.cc:407] Connection header received
0925 03:00:42.519662 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:42.519671 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:42.519906 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:42.520100 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:00:42.520731 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.522369 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.525190 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.526948 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.527113 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:42.527294 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:42.527843 (+ 549us) server_negotiation.cc:299] Negotiation successful
0925 03:00:42.528006 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":373,"threads_started":1}
I20260925 03:00:42.531788 3279 tablet_service.cc:1467] Processing CreateTablet for tablet c0e9a5d9ee3e40f5a642b1f72440ff38 (DEFAULT_TABLE table=test-table [id=6b2be39fcb604ab9be9ab48b82258f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 03:00:42.533234 3355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:42.513646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33491 (local address 127.0.0.1:51222)
0925 03:00:42.515011 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:42.515053 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 03:00:42.515238 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:00:42.519793 (+ 4555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:00:42.519798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:00:42.519833 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:00:42.520198 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:42.520213 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.522200 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.522204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:42.524616 (+ 2412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:42.524631 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.532066 (+ 7435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.532073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:42.532249 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:42.532294 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 03:00:42.532362 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 03:00:42.532484 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1093,"mutex_wait_us":82,"thread_start_us":945,"threads_started":1}
I20260925 03:00:42.533466 3279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0e9a5d9ee3e40f5a642b1f72440ff38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:00:42.534087 3356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:42.513795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51222 (local address 127.18.196.193:33491)
0925 03:00:42.519366 (+ 5571us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:42.519371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:42.519392 (+ 21us) server_negotiation.cc:407] Connection header received
0925 03:00:42.519463 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:42.519468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:42.519613 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:42.519798 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:00:42.520430 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.521996 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.524819 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.531781 (+ 6962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.533374 (+ 1593us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:42.533530 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:42.533603 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 03:00:42.533765 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5413,"thread_start_us":5308,"threads_started":1}
I20260925 03:00:42.534487 3148 tablet_service.cc:1467] Processing CreateTablet for tablet c0e9a5d9ee3e40f5a642b1f72440ff38 (DEFAULT_TABLE table=test-table [id=6b2be39fcb604ab9be9ab48b82258f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 03:00:42.535549 3148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0e9a5d9ee3e40f5a642b1f72440ff38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:00:42.537408 3015 tablet_service.cc:1467] Processing CreateTablet for tablet c0e9a5d9ee3e40f5a642b1f72440ff38 (DEFAULT_TABLE table=test-table [id=6b2be39fcb604ab9be9ab48b82258f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 03:00:42.538707 3015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0e9a5d9ee3e40f5a642b1f72440ff38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:00:42.547632 3361 tablet_bootstrap.cc:492] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae: Bootstrap starting.
I20260925 03:00:42.550654 3361 tablet_bootstrap.cc:654] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae: Neither blocks nor log segments found. Creating new log.
I20260925 03:00:42.551560 3362 tablet_bootstrap.cc:492] T c0e9a5d9ee3e40f5a642b1f72440ff38 P 3b5c6982fd6549ddb53e9a3aef215f1e: Bootstrap starting.
I20260925 03:00:42.552016 3361 log.cc:824] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae: Log is configured to *not* fsync() on all Append() calls
I20260925 03:00:42.554139 3364 tablet_bootstrap.cc:492] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87: Bootstrap starting.
I20260925 03:00:42.555425 3362 tablet_bootstrap.cc:654] T c0e9a5d9ee3e40f5a642b1f72440ff38 P 3b5c6982fd6549ddb53e9a3aef215f1e: Neither blocks nor log segments found. Creating new log.
I20260925 03:00:42.555804 3361 tablet_bootstrap.cc:492] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae: No bootstrap required, opened a new log
I20260925 03:00:42.556123 3361 ts_tablet_manager.cc:1397] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 03:00:42.556763 3362 log.cc:824] T c0e9a5d9ee3e40f5a642b1f72440ff38 P 3b5c6982fd6549ddb53e9a3aef215f1e: Log is configured to *not* fsync() on all Append() calls
I20260925 03:00:42.557176 3364 tablet_bootstrap.cc:654] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87: Neither blocks nor log segments found. Creating new log.
I20260925 03:00:42.558394 3364 log.cc:824] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87: Log is configured to *not* fsync() on all Append() calls
I20260925 03:00:42.559767 3362 tablet_bootstrap.cc:492] T c0e9a5d9ee3e40f5a642b1f72440ff38 P 3b5c6982fd6549ddb53e9a3aef215f1e: No bootstrap required, opened a new log
I20260925 03:00:42.560055 3362 ts_tablet_manager.cc:1397] T c0e9a5d9ee3e40f5a642b1f72440ff38 P 3b5c6982fd6549ddb53e9a3aef215f1e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 03:00:42.560871 3364 tablet_bootstrap.cc:492] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87: No bootstrap required, opened a new log
I20260925 03:00:42.561173 3364 ts_tablet_manager.cc:1397] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260925 03:00:42.561688 3080 heartbeater.cc:503] Master 127.18.196.254:46387 was elected leader, sending a full tablet report...
I20260925 03:00:42.566351 3361 raft_consensus.cc:348] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339037fb3c14611b0ca6c8892c83e87" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33491 } } peers { permanent_uuid: "f7362975190748bd837bc87ec6f9e5ae" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37227 } } peers { permanent_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 37589 } }
I20260925 03:00:42.566778 3361 raft_consensus.cc:374] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:00:42.566910 3361 raft_consensus.cc:729] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7362975190748bd837bc87ec6f9e5ae, State: Initialized, Role: FOLLOWER
I20260925 03:00:42.567512 3361 consensus_queue.cc:260] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339037fb3c14611b0ca6c8892c83e87" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33491 } } peers { permanent_uuid: "f7362975190748bd837bc87ec6f9e5ae" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37227 } } peers { permanent_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 37589 } }
I20260925 03:00:42.568614 3213 heartbeater.cc:503] Master 127.18.196.254:46387 was elected leader, sending a full tablet report...
I20260925 03:00:42.569185 3361 ts_tablet_manager.cc:1428] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 03:00:42.570158 3364 raft_consensus.cc:348] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339037fb3c14611b0ca6c8892c83e87" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33491 } } peers { permanent_uuid: "f7362975190748bd837bc87ec6f9e5ae" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37227 } } peers { permanent_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 37589 } }
I20260925 03:00:42.570639 3364 raft_consensus.cc:374] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:00:42.570807 3364 raft_consensus.cc:729] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a339037fb3c14611b0ca6c8892c83e87, State: Initialized, Role: FOLLOWER
I20260925 03:00:42.571548 3364 consensus_queue.cc:260] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339037fb3c14611b0ca6c8892c83e87" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33491 } } peers { permanent_uuid: "f7362975190748bd837bc87ec6f9e5ae" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37227 } } peers { permanent_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 37589 } }
I20260925 03:00:42.573256 3364 ts_tablet_manager.cc:1428] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260925 03:00:42.573447 3362 raft_consensus.cc:348] T c0e9a5d9ee3e40f5a642b1f72440ff38 P 3b5c6982fd6549ddb53e9a3aef215f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339037fb3c14611b0ca6c8892c83e87" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33491 } } peers { permanent_uuid: "f7362975190748bd837bc87ec6f9e5ae" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37227 } } peers { permanent_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 37589 } }
I20260925 03:00:42.573881 3362 raft_consensus.cc:374] T c0e9a5d9ee3e40f5a642b1f72440ff38 P 3b5c6982fd6549ddb53e9a3aef215f1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:00:42.573983 3362 raft_consensus.cc:729] T c0e9a5d9ee3e40f5a642b1f72440ff38 P 3b5c6982fd6549ddb53e9a3aef215f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5c6982fd6549ddb53e9a3aef215f1e, State: Initialized, Role: FOLLOWER
I20260925 03:00:42.574558 3362 consensus_queue.cc:260] T c0e9a5d9ee3e40f5a642b1f72440ff38 P 3b5c6982fd6549ddb53e9a3aef215f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339037fb3c14611b0ca6c8892c83e87" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33491 } } peers { permanent_uuid: "f7362975190748bd837bc87ec6f9e5ae" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37227 } } peers { permanent_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 37589 } }
I20260925 03:00:42.575526 3345 heartbeater.cc:503] Master 127.18.196.254:46387 was elected leader, sending a full tablet report...
I20260925 03:00:42.576220 3362 ts_tablet_manager.cc:1428] T c0e9a5d9ee3e40f5a642b1f72440ff38 P 3b5c6982fd6549ddb53e9a3aef215f1e: Time spent starting tablet: real 0.016s user 0.005s sys 0.008s
I20260925 03:00:42.632081 3367 raft_consensus.cc:482] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:00:42.632350 3367 raft_consensus.cc:504] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339037fb3c14611b0ca6c8892c83e87" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33491 } } peers { permanent_uuid: "f7362975190748bd837bc87ec6f9e5ae" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37227 } } peers { permanent_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 37589 } }
I20260925 03:00:42.634093 3367 leader_election.cc:288] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a339037fb3c14611b0ca6c8892c83e87 (127.18.196.193:33491), 3b5c6982fd6549ddb53e9a3aef215f1e (127.18.196.195:37589)
I20260925 03:00:42.641880 3358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:42.634652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:45367 (local address 127.18.196.195:37589)
0925 03:00:42.634926 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:42.634935 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:42.635439 (+ 504us) server_negotiation.cc:407] Connection header received
0925 03:00:42.635599 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:42.635604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:42.635687 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:42.635808 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:00:42.636315 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.637059 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.639677 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.640875 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.641039 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:42.641110 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:42.641482 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 03:00:42.641606 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 03:00:42.641983 3371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:42.634391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:37589 (local address 127.18.196.194:45367)
0925 03:00:42.635320 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:42.635343 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 03:00:42.635461 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:00:42.635814 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:00:42.635819 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:00:42.635843 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:00:42.636154 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:42.636162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.637210 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.637216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:42.639525 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:42.639539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.641082 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.641087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:42.641314 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:42.641349 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 03:00:42.641454 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 03:00:42.641586 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"thread_start_us":727,"threads_started":1}
I20260925 03:00:42.642028 3356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:42.634414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:48181 (local address 127.18.196.193:33491)
0925 03:00:42.634648 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:42.634657 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:42.635593 (+ 936us) server_negotiation.cc:407] Connection header received
0925 03:00:42.635754 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:42.635760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:42.635839 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:42.635927 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:00:42.636658 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.637510 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.639882 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.641074 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.641238 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:42.641295 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:42.641638 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 03:00:42.641784 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 03:00:42.642017 3370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:42.634432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33491 (local address 127.18.196.194:48181)
0925 03:00:42.635438 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:42.635463 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 03:00:42.635610 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:00:42.636107 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:00:42.636111 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:00:42.636131 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:00:42.636502 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:42.636509 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.637685 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.637695 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:42.639742 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:00:42.639752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.641302 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.641308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:00:42.641492 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:42.641521 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 03:00:42.641595 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 03:00:42.641719 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"thread_start_us":467,"threads_started":1}
I20260925 03:00:42.643167 3300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e9a5d9ee3e40f5a642b1f72440ff38" candidate_uuid: "f7362975190748bd837bc87ec6f9e5ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e" is_pre_election: true
I20260925 03:00:42.643187 3035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e9a5d9ee3e40f5a642b1f72440ff38" candidate_uuid: "f7362975190748bd837bc87ec6f9e5ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a339037fb3c14611b0ca6c8892c83e87" is_pre_election: true
I20260925 03:00:42.643647 3300 raft_consensus.cc:2454] T c0e9a5d9ee3e40f5a642b1f72440ff38 P 3b5c6982fd6549ddb53e9a3aef215f1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7362975190748bd837bc87ec6f9e5ae in term 0.
I20260925 03:00:42.643716 3035 raft_consensus.cc:2454] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7362975190748bd837bc87ec6f9e5ae in term 0.
I20260925 03:00:42.644356 3100 leader_election.cc:302] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5c6982fd6549ddb53e9a3aef215f1e, f7362975190748bd837bc87ec6f9e5ae; no voters:
I20260925 03:00:42.644799 3367 raft_consensus.cc:2781] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:00:42.644953 3367 raft_consensus.cc:482] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:00:42.645069 3367 raft_consensus.cc:3037] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:00:42.647262 3367 raft_consensus.cc:504] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339037fb3c14611b0ca6c8892c83e87" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33491 } } peers { permanent_uuid: "f7362975190748bd837bc87ec6f9e5ae" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37227 } } peers { permanent_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 37589 } }
I20260925 03:00:42.647800 3367 leader_election.cc:288] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae [CANDIDATE]: Term 1 election: Requested vote from peers a339037fb3c14611b0ca6c8892c83e87 (127.18.196.193:33491), 3b5c6982fd6549ddb53e9a3aef215f1e (127.18.196.195:37589)
I20260925 03:00:42.648169 3035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e9a5d9ee3e40f5a642b1f72440ff38" candidate_uuid: "f7362975190748bd837bc87ec6f9e5ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a339037fb3c14611b0ca6c8892c83e87"
I20260925 03:00:42.648232 3300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e9a5d9ee3e40f5a642b1f72440ff38" candidate_uuid: "f7362975190748bd837bc87ec6f9e5ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e"
I20260925 03:00:42.648403 3035 raft_consensus.cc:3037] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:00:42.648456 3300 raft_consensus.cc:3037] T c0e9a5d9ee3e40f5a642b1f72440ff38 P 3b5c6982fd6549ddb53e9a3aef215f1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:00:42.650552 3035 raft_consensus.cc:2454] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7362975190748bd837bc87ec6f9e5ae in term 1.
I20260925 03:00:42.650655 3300 raft_consensus.cc:2454] T c0e9a5d9ee3e40f5a642b1f72440ff38 P 3b5c6982fd6549ddb53e9a3aef215f1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7362975190748bd837bc87ec6f9e5ae in term 1.
I20260925 03:00:42.651046 3102 leader_election.cc:302] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a339037fb3c14611b0ca6c8892c83e87, f7362975190748bd837bc87ec6f9e5ae; no voters:
I20260925 03:00:42.651340 3367 raft_consensus.cc:2781] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:00:42.651785 3367 raft_consensus.cc:686] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae [term 1 LEADER]: Becoming Leader. State: Replica: f7362975190748bd837bc87ec6f9e5ae, State: Running, Role: LEADER
I20260925 03:00:42.652227 3367 consensus_queue.cc:237] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339037fb3c14611b0ca6c8892c83e87" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33491 } } peers { permanent_uuid: "f7362975190748bd837bc87ec6f9e5ae" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37227 } } peers { permanent_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 37589 } }
I20260925 03:00:42.656584 2893 catalog_manager.cc:5495] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae reported cstate change: term changed from 0 to 1, leader changed from <none> to f7362975190748bd837bc87ec6f9e5ae (127.18.196.194). New cstate: current_term: 1 leader_uuid: "f7362975190748bd837bc87ec6f9e5ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339037fb3c14611b0ca6c8892c83e87" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7362975190748bd837bc87ec6f9e5ae" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 37227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 37589 } health_report { overall_health: UNKNOWN } } }
I20260925 03:00:42.678392 3360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:42.673054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37944 (local address 127.18.196.194:37227)
0925 03:00:42.673244 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:42.673250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:42.673402 (+ 152us) server_negotiation.cc:407] Connection header received
0925 03:00:42.673632 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:42.673638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:42.673729 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:42.673873 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:00:42.674230 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.675006 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.676393 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.676797 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.676929 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:42.677414 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:00:42.677860 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:00:42.678004 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:42.678076 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 03:00:42.678144 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 03:00:42.686057 3300 raft_consensus.cc:1261] T c0e9a5d9ee3e40f5a642b1f72440ff38 P 3b5c6982fd6549ddb53e9a3aef215f1e [term 1 FOLLOWER]: Refusing update from remote peer f7362975190748bd837bc87ec6f9e5ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:00:42.686058 3035 raft_consensus.cc:1261] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87 [term 1 FOLLOWER]: Refusing update from remote peer f7362975190748bd837bc87ec6f9e5ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:00:42.687105 3372 consensus_queue.cc:1035] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a339037fb3c14611b0ca6c8892c83e87" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:00:42.687477 3367 consensus_queue.cc:1035] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5c6982fd6549ddb53e9a3aef215f1e" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 37589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 03:00:42.690976 3346 tablet.cc:2369] T c0e9a5d9ee3e40f5a642b1f72440ff38 P 3b5c6982fd6549ddb53e9a3aef215f1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 03:00:42.699574 3376 mvcc.cc:205] Tried to move back new op lower bound from 7333090274031878144 to 7333090273907650560. Current Snapshot: MvccSnapshot[applied={T|T < 7333090274031878144}]
I20260925 03:00:42.711404 2893 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 03:00:42.711664 2893 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 03:00:42.716063 2893 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb: Sending DeleteTablet for 3 replicas of tablet c0e9a5d9ee3e40f5a642b1f72440ff38
I20260925 03:00:42.717878 3279 tablet_service.cc:1514] Processing DeleteTablet for tablet c0e9a5d9ee3e40f5a642b1f72440ff38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 03:00:42 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:45796
I20260925 03:00:42.717913 3015 tablet_service.cc:1514] Processing DeleteTablet for tablet c0e9a5d9ee3e40f5a642b1f72440ff38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 03:00:42 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:51222
I20260925 03:00:42.718550 3148 tablet_service.cc:1514] Processing DeleteTablet for tablet c0e9a5d9ee3e40f5a642b1f72440ff38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 03:00:42 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:37942
I20260925 03:00:42.719099 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 2931
I20260925 03:00:42.729256 3385 tablet_replica.cc:330] T c0e9a5d9ee3e40f5a642b1f72440ff38 P 3b5c6982fd6549ddb53e9a3aef215f1e: stopping tablet replica
I20260925 03:00:42.729754 3385 raft_consensus.cc:2229] T c0e9a5d9ee3e40f5a642b1f72440ff38 P 3b5c6982fd6549ddb53e9a3aef215f1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:00:42.730229 3385 raft_consensus.cc:2258] T c0e9a5d9ee3e40f5a642b1f72440ff38 P 3b5c6982fd6549ddb53e9a3aef215f1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:00:42.730532 3384 tablet_replica.cc:330] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87: stopping tablet replica
I20260925 03:00:42.730940 3384 raft_consensus.cc:2229] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:00:42.731467 3384 raft_consensus.cc:2258] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:00:42.732597 3385 ts_tablet_manager.cc:1905] T c0e9a5d9ee3e40f5a642b1f72440ff38 P 3b5c6982fd6549ddb53e9a3aef215f1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 03:00:42.735352 3384 ts_tablet_manager.cc:1905] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 03:00:42.739493 3385 ts_tablet_manager.cc:1918] T c0e9a5d9ee3e40f5a642b1f72440ff38 P 3b5c6982fd6549ddb53e9a3aef215f1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 03:00:42.739619 3356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:42.720202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51226 (local address 127.18.196.193:33491)
0925 03:00:42.720767 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:42.720773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:42.720790 (+ 17us) server_negotiation.cc:407] Connection header received
0925 03:00:42.720850 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:42.720857 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:42.720958 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:42.721067 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:00:42.723900 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.725350 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.727701 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.728508 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.728634 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:42.732483 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:00:42.732499 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:00:42.732517 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:00:42.732565 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:00:42.736383 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:42.737289 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:42.737302 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:42.737306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:42.737447 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:42.737850 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:42.737855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:42.737859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:42.738337 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:00:42.738610 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:42.738995 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 03:00:42.739243 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444}
I20260925 03:00:42.739784 3385 log.cc:1196] T c0e9a5d9ee3e40f5a642b1f72440ff38 P 3b5c6982fd6549ddb53e9a3aef215f1e: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/c0e9a5d9ee3e40f5a642b1f72440ff38
I20260925 03:00:42.740361 3385 ts_tablet_manager.cc:1939] T c0e9a5d9ee3e40f5a642b1f72440ff38 P 3b5c6982fd6549ddb53e9a3aef215f1e: Deleting consensus metadata
I20260925 03:00:42.740507 3075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:00:42.742208 2877 catalog_manager.cc:4841] TS 3b5c6982fd6549ddb53e9a3aef215f1e (127.18.196.195:37589): tablet c0e9a5d9ee3e40f5a642b1f72440ff38 (table test-table [id=6b2be39fcb604ab9be9ab48b82258f40]) successfully deleted
I20260925 03:00:42.743353 3386 tablet_replica.cc:330] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae: stopping tablet replica
I20260925 03:00:42.743793 3386 raft_consensus.cc:2229] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae [term 1 LEADER]: Raft consensus shutting down.
I20260925 03:00:42.744414 3386 raft_consensus.cc:2258] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:00:42.748090 3386 ts_tablet_manager.cc:1905] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 03:00:42.753036 3386 ts_tablet_manager.cc:1918] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 03:00:42.753247 3386 log.cc:1196] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/c0e9a5d9ee3e40f5a642b1f72440ff38
I20260925 03:00:42.753746 3386 ts_tablet_manager.cc:1939] T c0e9a5d9ee3e40f5a642b1f72440ff38 P f7362975190748bd837bc87ec6f9e5ae: Deleting consensus metadata
I20260925 03:00:42.755283 2879 catalog_manager.cc:4841] TS f7362975190748bd837bc87ec6f9e5ae (127.18.196.194:37227): tablet c0e9a5d9ee3e40f5a642b1f72440ff38 (table test-table [id=6b2be39fcb604ab9be9ab48b82258f40]) successfully deleted
I20260925 03:00:42.839311 3384 ts_tablet_manager.cc:1918] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 03:00:42.839593 3384 log.cc:1196] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/wal/wals/c0e9a5d9ee3e40f5a642b1f72440ff38
I20260925 03:00:42.840131 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 2931
I20260925 03:00:42.840219 3384 ts_tablet_manager.cc:1939] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87: Deleting consensus metadata
W20260925 03:00:42.858579 2879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.196.193:33491 (error 108)
I20260925 03:00:42.858899 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 3084
I20260925 03:00:42.860899 3355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:42.860392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33491 (local address 127.0.0.1:51238)
0925 03:00:42.860591 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:42.860722 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:33491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260925 03:00:42.861182 2879 catalog_manager.cc:4584] TS a339037fb3c14611b0ca6c8892c83e87 (127.18.196.193:33491): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c0e9a5d9ee3e40f5a642b1f72440ff38: Network error: Client connection negotiation failed: client connection to 127.18.196.193:33491: connect: Connection refused (error 111)
I20260925 03:00:42.875061 3360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:42.859909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37960 (local address 127.18.196.194:37227)
0925 03:00:42.860084 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:42.860088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:42.860108 (+ 20us) server_negotiation.cc:407] Connection header received
0925 03:00:42.860265 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:42.860270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:42.860369 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:42.860481 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:00:42.862637 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.863763 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.865259 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:42.865703 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:42.865844 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:42.868629 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:00:42.868642 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:00:42.868656 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:00:42.868695 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:00:42.872037 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:42.872750 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:42.872759 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:42.872762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:42.872918 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:42.873376 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:42.873381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:42.873385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:42.873719 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:00:42.874082 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:42.874415 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 03:00:42.874645 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 03:00:42.875829 3208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:00:42.914788 3355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:42.914224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33491 (local address 127.0.0.1:51248)
0925 03:00:42.914472 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:42.914583 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:33491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 03:00:42.915903 3355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:42.915507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33491 (local address 127.0.0.1:51258)
0925 03:00:42.915663 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:42.915764 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:33491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 03:00:42.976475 3355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:42.975932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33491 (local address 127.0.0.1:51260)
0925 03:00:42.976173 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:42.976293 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:33491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 03:00:42.977563 3355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:42.977166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33491 (local address 127.0.0.1:51266)
0925 03:00:42.977338 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:42.977421 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:33491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 03:00:42.991272 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 3084
I20260925 03:00:43.011382 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 3216
I20260925 03:00:43.029783 3358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:43.012117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45804 (local address 127.18.196.195:37589)
0925 03:00:43.012289 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 03:00:43.012294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:00:43.012311 (+ 17us) server_negotiation.cc:407] Connection header received
0925 03:00:43.012458 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:00:43.012462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:00:43.012526 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:00:43.012611 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:00:43.014078 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:43.014814 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:43.016166 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:00:43.016627 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:00:43.016724 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:00:43.023933 (+ 7209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:00:43.023945 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:00:43.023961 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:00:43.023999 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:00:43.026975 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:43.027630 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:43.027638 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:43.027640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:43.027818 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:00:43.028232 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:00:43.028239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:00:43.028246 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:00:43.028572 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:00:43.028856 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 03:00:43.029234 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 03:00:43.029459 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 03:00:43.030406 3340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:00:43.058974 3355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:43.058221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33491 (local address 127.0.0.1:51270)
0925 03:00:43.058442 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:43.058748 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:33491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 03:00:43.060153 3355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:43.059658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33491 (local address 127.0.0.1:51284)
0925 03:00:43.059892 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:43.059996 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:33491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260925 03:00:43.140604 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 3216
I20260925 03:00:43.157078 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 2860
I20260925 03:00:43.158196 2924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:00:43.272819 3355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:00:43.272286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33491 (local address 127.0.0.1:51300)
0925 03:00:43.272503 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 03:00:43.272612 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:33491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 03:00:43.273208 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 2860
I20260925 03:01:03.295203 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:46387
--webserver_interface=127.18.196.254
--webserver_port=35321
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:46387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:01:03.612589 3392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:03.612907 3392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:03.613057 3392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:03.619956 3392 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 03:01:03.620062 3392 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 03:01:03.620137 3392 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 03:01:03.620204 3392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:01:03.620273 3392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:03.620359 3392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:03.620441 3392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:01:03.620510 3392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:01:03.629545 3392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:46387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:46387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=35321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:03.630748 3392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:03.632009 3392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:03.643378 3397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:03.643406 3398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:03.643347 3400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:03.643994 3392 server_base.cc:1034] running on GCE node
I20260925 03:01:03.644624 3392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:03.645759 3392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:03.647010 3392 hybrid_clock.cc:648] HybridClock initialized: now 1790305263646955 us; error 55 us; skew 500 ppm
I20260925 03:01:03.647317 3392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:03 dist-test-slave-8zvk krb5kdc[2303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305263, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:03.650096 3392 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 03:01:03.651404 3392 webserver.cc:460] Webserver started at http://127.18.196.254:35321/ using document root <none> and password file <none>
I20260925 03:01:03.652015 3392 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:03.652139 3392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:03.657042 3392 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 03:01:03.659508 3407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:03.660614 3392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 03:01:03.660768 3392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "89ae4c9f9042478499d8c6a40f3cc0bb"
format_stamp: "Formatted at 2026-09-25 03:00:18 on dist-test-slave-8zvk"
server_key: "880f2ee6192cb43171778870400ebfd1"
server_key_iv: "ceb103f29afb2173dde1e580916dca46"
server_key_version: "encryptionkey@0"
I20260925 03:01:03.661746 3392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:03.676915 3392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:03.680313 3392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:03.680563 3392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:03.697878 3392 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:46387
I20260925 03:01:03.697949 3458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:46387 every 8 connection(s)
I20260925 03:01:03.699257 3392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:01:03.705430 3459 sys_catalog.cc:263] Verifying existing consensus state
I20260925 03:01:03.707356 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 3392
I20260925 03:01:03.708159 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:33491
--local_ip_for_outbound_sockets=127.18.196.193
--tserver_master_addrs=127.18.196.254:46387
--webserver_port=43231
--webserver_interface=127.18.196.193
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 03:01:03.708624 3459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb: Bootstrap starting.
I20260925 03:01:03.733174 3459 log.cc:824] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:03.753198 3459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 03:01:03.753742 3459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb: Bootstrap complete.
I20260925 03:01:03.763149 3459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae4c9f9042478499d8c6a40f3cc0bb" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 46387 } }
I20260925 03:01:03.764012 3459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89ae4c9f9042478499d8c6a40f3cc0bb, State: Initialized, Role: FOLLOWER
I20260925 03:01:03.764660 3459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae4c9f9042478499d8c6a40f3cc0bb" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 46387 } }
I20260925 03:01:03.764916 3459 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:01:03.765040 3459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:01:03.765221 3459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 2 FOLLOWER]: Advancing to term 3
I20260925 03:01:03.768990 3459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae4c9f9042478499d8c6a40f3cc0bb" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 46387 } }
I20260925 03:01:03.769464 3459 leader_election.cc:302] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89ae4c9f9042478499d8c6a40f3cc0bb; no voters:
I20260925 03:01:03.770184 3459 leader_election.cc:288] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 03:01:03.770321 3463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 3 FOLLOWER]: Leader election won for term 3
I20260925 03:01:03.771692 3463 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [term 3 LEADER]: Becoming Leader. State: Replica: 89ae4c9f9042478499d8c6a40f3cc0bb, State: Running, Role: LEADER
I20260925 03:01:03.772300 3463 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae4c9f9042478499d8c6a40f3cc0bb" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 46387 } }
I20260925 03:01:03.772455 3459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:01:03.774958 3465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89ae4c9f9042478499d8c6a40f3cc0bb. Latest consensus state: current_term: 3 leader_uuid: "89ae4c9f9042478499d8c6a40f3cc0bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae4c9f9042478499d8c6a40f3cc0bb" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 46387 } } }
I20260925 03:01:03.775383 3465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:03.776031 3464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "89ae4c9f9042478499d8c6a40f3cc0bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae4c9f9042478499d8c6a40f3cc0bb" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 46387 } } }
I20260925 03:01:03.776394 3464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:03.780568 3469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:01:03.788846 3469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:01:03.790210 3469 catalog_manager.cc:1259] Loaded cluster ID: fb3e3f55110242f18de7b693acb25141
I20260925 03:01:03.790357 3469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:01:03.794394 3469 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:01:03.795984 3469 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb: Loaded TSK: 0
I20260925 03:01:03.796197 3469 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 89ae4c9f9042478499d8c6a40f3cc0bb: Loaded TSK: 1
F20260925 03:01:03.797299 3469 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7e8806a9ad google::LogMessage::Fail() at ??:0
@ 0x7f7e8806d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f7e8806a4de google::LogMessage::Flush() at ??:0
@ 0x7f7e8806ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7e8bdce7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7e8bdbd73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f7e9da991c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f7e9da98783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f7e9db242a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f7e9db240c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f7e9db6e5e9 std::function<>::operator()() at ??:0
@ 0x7f7e9da9ccdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7e9da9ac36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7e9db22610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f7e9db22439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7e8acb6345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7e8acb991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f7e8acb9799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7e8ac94eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7e8df6e6db start_thread at ??:0
@ 0x7f7e86f0461f clone at ??:0
W20260925 03:01:04.106004 3461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:04.106312 3461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:04.106485 3461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:04.113374 3461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:04.113519 3461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:04.113701 3461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 03:01:04.122509 3461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:33491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=43231
--tserver_master_addrs=127.18.196.254:46387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:04.123719 3461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:04.124965 3461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:04.137396 3509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:04.137364 3507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:04.137481 3506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:04.138140 3461 server_base.cc:1034] running on GCE node
I20260925 03:01:04.138680 3461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:04.139572 3461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:04.140789 3461 hybrid_clock.cc:648] HybridClock initialized: now 1790305264140739 us; error 59 us; skew 500 ppm
I20260925 03:01:04.141099 3461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:04 dist-test-slave-8zvk krb5kdc[2303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305264, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:04.144057 3461 init.cc:377] Logged in from keytab as kudu/127.18.196.193@KRBTEST.COM (short username kudu)
I20260925 03:01:04.145468 3461 webserver.cc:460] Webserver started at http://127.18.196.193:43231/ using document root <none> and password file <none>
I20260925 03:01:04.146121 3461 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:04.146263 3461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:04.151211 3461 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 03:01:04.153685 3516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:04.154965 3461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 03:01:04.155145 3461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "a339037fb3c14611b0ca6c8892c83e87"
format_stamp: "Formatted at 2026-09-25 03:00:18 on dist-test-slave-8zvk"
server_key: "aefd6e2d7f6c33e9f1bd04ffad4905c4"
server_key_iv: "bab59001ce0220117fe1ed39b6094f19"
server_key_version: "encryptionkey@0"
I20260925 03:01:04.156212 3461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:04.184341 3461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:04.188093 3461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:04.188364 3461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:04.189297 3461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:01:04.190958 3523 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260925 03:01:04.195852 3523 ts_tablet_manager.cc:1852] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260925 03:01:04.196805 3523 ts_tablet_manager.cc:1905] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 03:01:04.203796 3523 ts_tablet_manager.cc:1918] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 03:01:04.204093 3523 ts_tablet_manager.cc:1939] T c0e9a5d9ee3e40f5a642b1f72440ff38 P a339037fb3c14611b0ca6c8892c83e87: Deleting consensus metadata
I20260925 03:01:04.204692 3461 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260925 03:01:04.204828 3461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.000s sys 0.000s
I20260925 03:01:04.204967 3461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:01:04.205060 3461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:04.238600 3461 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:33491
I20260925 03:01:04.238654 3629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:33491 every 8 connection(s)
I20260925 03:01:04.239904 3461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 03:01:04.244563 3632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:04.242747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.193:60757)
0925 03:01:04.243545 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:04.243760 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":282,"threads_started":1}
W20260925 03:01:04.245327 3630 heartbeater.cc:644] Failed to heartbeat to 127.18.196.254:46387 (0 consecutive failures): Network error: Failed to ping master at 127.18.196.254:46387: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
I20260925 03:01:04.246140 3632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:04.245814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.193:44945)
0925 03:01:04.245971 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:04.246045 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 03:01:04.247087 3632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:04.246650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.193:52399)
0925 03:01:04.246812 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:04.246933 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 03:01:04.247284 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 3461
W20260925 03:01:04.247381 3630 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 03:01:04.248003 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:37227
--local_ip_for_outbound_sockets=127.18.196.194
--tserver_master_addrs=127.18.196.254:46387
--webserver_port=33743
--webserver_interface=127.18.196.194
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:01:04.564426 3633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:04.564704 3633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:04.564798 3633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:04.572001 3633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:04.572113 3633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:04.572228 3633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 03:01:04.581831 3633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:37227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=33743
--tserver_master_addrs=127.18.196.254:46387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:04.583189 3633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:04.584451 3633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:04.597329 3641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:04.597425 3639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:04.597448 3638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:04.598519 3633 server_base.cc:1034] running on GCE node
I20260925 03:01:04.599200 3633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:04.600162 3633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:04.601416 3633 hybrid_clock.cc:648] HybridClock initialized: now 1790305264601366 us; error 40 us; skew 500 ppm
I20260925 03:01:04.601790 3633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:04 dist-test-slave-8zvk krb5kdc[2303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305264, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:04.604924 3633 init.cc:377] Logged in from keytab as kudu/127.18.196.194@KRBTEST.COM (short username kudu)
I20260925 03:01:04.606396 3633 webserver.cc:460] Webserver started at http://127.18.196.194:33743/ using document root <none> and password file <none>
I20260925 03:01:04.607179 3633 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:04.607368 3633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:04.613912 3633 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260925 03:01:04.616709 3648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:04.618142 3633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 03:01:04.618321 3633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "f7362975190748bd837bc87ec6f9e5ae"
format_stamp: "Formatted at 2026-09-25 03:00:19 on dist-test-slave-8zvk"
server_key: "623b0f2782b866c6d9a49f0a302d0c57"
server_key_iv: "c9826eebbf8b47ede715b069130ec636"
server_key_version: "encryptionkey@0"
I20260925 03:01:04.619426 3633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:04.657615 3633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:04.662048 3633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:04.662352 3633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:04.663386 3633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:01:04.664733 3633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:01:04.664862 3633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:04.665004 3633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:01:04.665096 3633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:04.698025 3633 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:37227
I20260925 03:01:04.698109 3760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:37227 every 8 connection(s)
I20260925 03:01:04.699517 3633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 03:01:04.701928 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 3633
I20260925 03:01:04.702674 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:37589
--local_ip_for_outbound_sockets=127.18.196.195
--tserver_master_addrs=127.18.196.254:46387
--webserver_port=40497
--webserver_interface=127.18.196.195
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 03:01:04.704559 3763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:04.702697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.194:55895)
0925 03:01:04.703507 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:04.703734 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":292,"threads_started":1}
W20260925 03:01:04.705227 3761 heartbeater.cc:644] Failed to heartbeat to 127.18.196.254:46387 (0 consecutive failures): Network error: Failed to ping master at 127.18.196.254:46387: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
I20260925 03:01:04.706043 3763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:04.705750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.194:52253)
0925 03:01:04.705885 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:04.705947 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 03:01:04.706972 3763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:04.706616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.194:53067)
0925 03:01:04.706741 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:04.706810 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 03:01:04.707244 3761 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 03:01:05.022723 3764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:05.023047 3764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:05.023243 3764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:05.030153 3764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:05.030239 3764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:05.030421 3764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 03:01:05.039404 3764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:37589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=40497
--tserver_master_addrs=127.18.196.254:46387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:05.040521 3764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:05.041723 3764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:05.054193 3770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:05.054296 3769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:05.054273 3772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:05.055073 3764 server_base.cc:1034] running on GCE node
I20260925 03:01:05.055631 3764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:05.056461 3764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:05.057677 3764 hybrid_clock.cc:648] HybridClock initialized: now 1790305265057639 us; error 45 us; skew 500 ppm
I20260925 03:01:05.057968 3764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:05 dist-test-slave-8zvk krb5kdc[2303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305265, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:05.061107 3764 init.cc:377] Logged in from keytab as kudu/127.18.196.195@KRBTEST.COM (short username kudu)
I20260925 03:01:05.062523 3764 webserver.cc:460] Webserver started at http://127.18.196.195:40497/ using document root <none> and password file <none>
I20260925 03:01:05.063248 3764 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:05.063395 3764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:05.068351 3764 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260925 03:01:05.070820 3779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:05.072093 3764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 03:01:05.072263 3764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "3b5c6982fd6549ddb53e9a3aef215f1e"
format_stamp: "Formatted at 2026-09-25 03:00:19 on dist-test-slave-8zvk"
server_key: "c3d41831a73b82a666f4636dbef811bb"
server_key_iv: "6c213b760fb3411753b4f2f5602f6a17"
server_key_version: "encryptionkey@0"
I20260925 03:01:05.073216 3764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:05.087109 3764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:05.091763 3764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:05.092039 3764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:05.093031 3764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:01:05.094306 3764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:01:05.094441 3764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:05.094563 3764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:01:05.094651 3764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:05.128628 3764 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:37589
I20260925 03:01:05.128726 3891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:37589 every 8 connection(s)
I20260925 03:01:05.130095 3764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 03:01:05.134627 3894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:05.132871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.195:53405)
0925 03:01:05.133608 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:05.133832 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":312,"threads_started":1}
W20260925 03:01:05.135349 3892 heartbeater.cc:644] Failed to heartbeat to 127.18.196.254:46387 (0 consecutive failures): Network error: Failed to ping master at 127.18.196.254:46387: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
I20260925 03:01:05.136134 3894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:05.135798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.195:35501)
0925 03:01:05.135956 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:05.136030 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 03:01:05.136940 3894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:05.136625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.195:60519)
0925 03:01:05.136758 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:05.136830 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 03:01:05.137161 3892 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 03:01:05.137662 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 3764
I20260925 03:01:05.248687 3896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:05.247879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.193:44671)
0925 03:01:05.248396 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:05.248499 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":287,"threads_started":1}
I20260925 03:01:05.708727 3897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:05.707774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.194:46979)
0925 03:01:05.708289 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:05.708418 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":267,"threads_started":1}
I20260925 03:01:06.138453 3898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:06.137715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.195:57735)
0925 03:01:06.138141 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:06.138234 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":223,"threads_started":1}
I20260925 03:01:06.250227 3899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:06.249524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.193:45201)
0925 03:01:06.249952 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:06.250037 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":241,"threads_started":1}
I20260925 03:01:06.710322 3900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:06.709676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.194:47993)
0925 03:01:06.710092 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:06.710169 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":210,"threads_started":1}
I20260925 03:01:07.139936 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:07.139276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.195:41033)
0925 03:01:07.139715 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:07.139788 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":227,"threads_started":1}
I20260925 03:01:07.251895 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:07.251105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.193:40869)
0925 03:01:07.251628 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:07.251715 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":298,"threads_started":1}
I20260925 03:01:07.711859 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:07.711212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.194:54953)
0925 03:01:07.711613 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:07.711688 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":212,"threads_started":1}
I20260925 03:01:08.141441 3904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:08.140749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.195:56579)
0925 03:01:08.141205 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:08.141281 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":258,"threads_started":1}
I20260925 03:01:08.253693 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:08.252912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.193:37409)
0925 03:01:08.253439 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:08.253517 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":300,"threads_started":1}
I20260925 03:01:08.713678 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:08.712968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.194:60517)
0925 03:01:08.713430 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:08.713514 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":231,"threads_started":1}
I20260925 03:01:09.143030 3907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:09.142291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.195:35195)
0925 03:01:09.142695 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:09.142770 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":190,"threads_started":1}
I20260925 03:01:09.255510 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:09.254644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.193:58587)
0925 03:01:09.255213 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:09.255313 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":327,"threads_started":1}
I20260925 03:01:09.715420 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:09.714640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.194:48147)
0925 03:01:09.715155 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:09.715239 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":296,"threads_started":1}
I20260925 03:01:10.144784 3910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:10.144079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.195:38369)
0925 03:01:10.144531 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:10.144616 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":215,"threads_started":1}
I20260925 03:01:10.257263 3911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:10.256452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.193:39607)
0925 03:01:10.256998 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:10.257092 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":308,"threads_started":1}
I20260925 03:01:10.716951 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:10.716243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.194:40385)
0925 03:01:10.716682 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:10.716791 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":225,"threads_started":1}
I20260925 03:01:11.146484 3913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:11.145792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.195:49383)
0925 03:01:11.146228 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:11.146326 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":234,"threads_started":1}
I20260925 03:01:11.259203 3914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:11.258283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.193:54547)
0925 03:01:11.258902 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:11.259007 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":298,"threads_started":1}
I20260925 03:01:11.718565 3915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:11.717793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.194:59563)
0925 03:01:11.718309 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:11.718384 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":245,"threads_started":1}
I20260925 03:01:12.148172 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:12.147369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.195:48795)
0925 03:01:12.147860 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:12.147955 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":226,"threads_started":1}
I20260925 03:01:12.260739 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:12.260018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.193:56725)
0925 03:01:12.260482 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:12.260568 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":270,"threads_started":1}
I20260925 03:01:12.720139 3918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:12.719480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.194:39401)
0925 03:01:12.719897 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:12.719980 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":236,"threads_started":1}
I20260925 03:01:13.149910 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:13.149203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.195:33263)
0925 03:01:13.149663 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:13.149743 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":259,"threads_started":1}
I20260925 03:01:13.262235 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:13.261537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.193:38061)
0925 03:01:13.262003 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:13.262094 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":238,"threads_started":1}
I20260925 03:01:13.721742 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:13.720908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.194:43431)
0925 03:01:13.721473 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:13.721563 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":314,"threads_started":1}
I20260925 03:01:14.151561 3922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:14.150770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.195:60471)
0925 03:01:14.151305 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:14.151390 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":295,"threads_started":1}
I20260925 03:01:14.263876 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:14.263171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.193:41283)
0925 03:01:14.263602 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:14.263686 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":223,"threads_started":1}
I20260925 03:01:14.723393 3924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:14.722695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.194:36179)
0925 03:01:14.723155 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:14.723237 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":270,"threads_started":1}
I20260925 03:01:15.153154 3925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:15.152432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.195:35811)
0925 03:01:15.152893 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:15.152971 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":265,"threads_started":1}
I20260925 03:01:15.265606 3926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:15.264795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.193:58043)
0925 03:01:15.265347 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:15.265430 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":343,"threads_started":1}
I20260925 03:01:15.724943 3927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:15.724236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.194:39737)
0925 03:01:15.724659 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:15.724753 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":205,"threads_started":1}
I20260925 03:01:16.154681 3928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:16.153977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.195:59329)
0925 03:01:16.154417 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:16.154505 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":219,"threads_started":1}
I20260925 03:01:16.267231 3929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:16.266500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.193:41567)
0925 03:01:16.266973 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:16.267058 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":217,"threads_started":1}
I20260925 03:01:16.726363 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:16.725735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.194:49475)
0925 03:01:16.726141 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:16.726220 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":197,"threads_started":1}
I20260925 03:01:17.156221 3931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:17.155554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.195:46349)
0925 03:01:17.155986 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:17.156073 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":237,"threads_started":1}
I20260925 03:01:17.268803 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:17.268060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.193:46955)
0925 03:01:17.268518 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:17.268622 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":236,"threads_started":1}
I20260925 03:01:17.728013 3933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:17.727291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.194:43839)
0925 03:01:17.727720 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:17.727809 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":220,"threads_started":1}
I20260925 03:01:18.157763 3934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:18.157064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.195:59067)
0925 03:01:18.157511 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:18.157593 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":231,"threads_started":1}
I20260925 03:01:18.270474 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:18.269766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.193:50057)
0925 03:01:18.270204 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:18.270286 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":225,"threads_started":1}
I20260925 03:01:18.729461 3936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:18.728781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.194:45311)
0925 03:01:18.729177 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:18.729250 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":199,"threads_started":1}
I20260925 03:01:19.159258 3937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:19.158572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.195:58525)
0925 03:01:19.159033 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:19.159103 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":279,"threads_started":1}
I20260925 03:01:19.272163 3938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:19.271458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.193:58007)
0925 03:01:19.271904 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:19.271997 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":212,"threads_started":1}
I20260925 03:01:19.731204 3939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:19.730460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.194:40581)
0925 03:01:19.730933 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:19.731029 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":209,"threads_started":1}
I20260925 03:01:20.139484 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 3461
I20260925 03:01:20.161206 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:20.160431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:46387 (local address 127.18.196.195:40631)
0925 03:01:20.160937 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:20.161029 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.254:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":271,"threads_started":1}
I20260925 03:01:20.162449 3940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:20.140470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33870 (local address 127.18.196.193:33491)
0925 03:01:20.141113 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:20.141136 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:20.141188 (+ 52us) server_negotiation.cc:407] Connection header received
0925 03:01:20.143183 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:20.143193 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:20.143617 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:20.143901 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:20.145229 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:20.146426 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:20.147739 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:20.148407 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:20.148539 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:20.155899 (+ 7360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:20.155907 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:20.155922 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:20.155960 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:20.158941 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:20.159754 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:20.159765 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:20.159773 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:20.160004 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:20.160464 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:20.160467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:20.160469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:20.160993 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:20.161299 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:20.161754 (+ 455us) server_negotiation.cc:299] Negotiation successful
0925 03:01:20.162171 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":287,"threads_started":1}
I20260925 03:01:20.163815 3625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:20.248190 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 3461
I20260925 03:01:20.265461 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 3633
I20260925 03:01:20.283140 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:20.266448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49214 (local address 127.18.196.194:37227)
0925 03:01:20.267032 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:20.267045 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:20.267086 (+ 41us) server_negotiation.cc:407] Connection header received
0925 03:01:20.268670 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:20.268679 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:20.269086 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:20.269430 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:20.270779 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:20.272136 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:20.273342 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:20.274002 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:20.274193 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:20.276849 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:20.276856 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:20.276869 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:20.276904 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:20.279823 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:20.280499 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:20.280507 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:20.280513 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:20.280669 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:20.281035 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:20.281041 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:20.281044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:20.281517 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:20.281851 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:20.282299 (+ 448us) server_negotiation.cc:299] Negotiation successful
0925 03:01:20.282695 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":251,"threads_started":1}
I20260925 03:01:20.284718 3756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:20.366099 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 3633
I20260925 03:01:20.383934 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 3764
I20260925 03:01:20.400686 3945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:20.385006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33466 (local address 127.18.196.195:37589)
0925 03:01:20.385669 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:20.385682 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:20.385720 (+ 38us) server_negotiation.cc:407] Connection header received
0925 03:01:20.387407 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:20.387419 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:20.387848 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:20.388116 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:20.389341 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:20.390538 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:20.391849 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:20.392459 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:20.392593 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:20.395061 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:20.395068 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:20.395081 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:20.395113 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:20.397865 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:20.398446 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:20.398452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:20.398456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:20.398606 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:20.399168 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:20.399171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:20.399173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:20.399595 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:20.399829 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:20.400133 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 03:01:20.400413 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":312,"threads_started":1}
I20260925 03:01:20.401934 3887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:20.483287 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 3764
2026-09-25T03:01:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64004 ms)
[----------] 30 tests from SecurityITest (292771 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 03:01:20 dist-test-slave-8zvk krb5kdc[3949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 03:01:20 dist-test-slave-8zvk krb5kdc[3949](info): set up 2 sockets
Sep 25 03:01:20 dist-test-slave-8zvk krb5kdc[3949](info): commencing operation
krb5kdc: starting...
W20260925 03:01:21.574637 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 03:01:21 dist-test-slave-8zvk krb5kdc[3949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T03:01:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:01:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 03:01:21.814554 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:42135
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:42135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:01:22.139178 3962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:22.139506 3962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:22.139679 3962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:22.146699 3962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:01:22.146890 3962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:22.146986 3962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:22.147074 3962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:01:22.147164 3962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:01:22.156780 3962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:42135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:42135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:22.158084 3962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:22.159360 3962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:22.171221 3968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:22.171304 3967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:22.171376 3970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:22.171856 3962 server_base.cc:1034] running on GCE node
I20260925 03:01:22.172597 3962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:22.173847 3962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:22.175037 3962 hybrid_clock.cc:648] HybridClock initialized: now 1790305282174974 us; error 54 us; skew 500 ppm
I20260925 03:01:22.175330 3962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:22 dist-test-slave-8zvk krb5kdc[3949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305282, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:22.178201 3962 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 03:01:22.179641 3962 webserver.cc:460] Webserver started at http://127.18.196.254:39057/ using document root <none> and password file <none>
I20260925 03:01:22.180294 3962 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:22.180433 3962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:22.180732 3962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:22.183678 3962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "ea9bc3b297404de68963d3dd25cd9b9a"
format_stamp: "Formatted at 2026-09-25 03:01:22 on dist-test-slave-8zvk"
server_key: "34fbc2682056a03f3f3adefe05a20a82"
server_key_iv: "92c0fcd7e9eb6a3a32d5e6f0a24751f5"
server_key_version: "encryptionkey@0"
I20260925 03:01:22.184419 3962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "ea9bc3b297404de68963d3dd25cd9b9a"
format_stamp: "Formatted at 2026-09-25 03:01:22 on dist-test-slave-8zvk"
server_key: "34fbc2682056a03f3f3adefe05a20a82"
server_key_iv: "92c0fcd7e9eb6a3a32d5e6f0a24751f5"
server_key_version: "encryptionkey@0"
I20260925 03:01:22.189250 3962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 03:01:22.192574 3977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:22.193758 3962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 03:01:22.193936 3962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "ea9bc3b297404de68963d3dd25cd9b9a"
format_stamp: "Formatted at 2026-09-25 03:01:22 on dist-test-slave-8zvk"
server_key: "34fbc2682056a03f3f3adefe05a20a82"
server_key_iv: "92c0fcd7e9eb6a3a32d5e6f0a24751f5"
server_key_version: "encryptionkey@0"
I20260925 03:01:22.194118 3962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:22.206432 3962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:22.209990 3962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:22.210256 3962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:22.227847 3962 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:42135
I20260925 03:01:22.227866 4028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:42135 every 8 connection(s)
I20260925 03:01:22.229059 3962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:01:22.232298 4029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:22.235360 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 3962
I20260925 03:01:22.235575 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:01:22.236215 19219 external_mini_cluster.cc:1428] Setting key 1ed1e8420a7c8a151510f4d42f8820a8
I20260925 03:01:22.244203 4029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9bc3b297404de68963d3dd25cd9b9a: Bootstrap starting.
I20260925 03:01:22.247704 4029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea9bc3b297404de68963d3dd25cd9b9a: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:22.248868 4029 log.cc:824] T 00000000000000000000000000000000 P ea9bc3b297404de68963d3dd25cd9b9a: Log is configured to *not* fsync() on all Append() calls
Sep 25 03:01:22 dist-test-slave-8zvk krb5kdc[3949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:01:22.251432 4029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9bc3b297404de68963d3dd25cd9b9a: No bootstrap required, opened a new log
I20260925 03:01:22.258949 4032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:22.238003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35208 (local address 127.18.196.254:42135)
0925 03:01:22.238733 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:22.238743 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:22.238778 (+ 35us) server_negotiation.cc:407] Connection header received
0925 03:01:22.240283 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:22.240301 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:22.240620 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:22.240893 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:22.242747 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:22.244021 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:22.245901 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:22.246541 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:22.246669 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:22.250977 (+ 4308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:22.250986 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:22.251003 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:22.251042 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:22.254661 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:22.255401 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:22.255409 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:22.255416 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:22.255631 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:22.256029 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:22.256035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:22.256051 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:22.256693 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:22.256991 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:22.257312 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 03:01:22.257707 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":352,"threads_started":1}
I20260925 03:01:22.261598 4029 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea9bc3b297404de68963d3dd25cd9b9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9bc3b297404de68963d3dd25cd9b9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 42135 } }
I20260925 03:01:22.261952 4029 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea9bc3b297404de68963d3dd25cd9b9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:22.262028 4029 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea9bc3b297404de68963d3dd25cd9b9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9bc3b297404de68963d3dd25cd9b9a, State: Initialized, Role: FOLLOWER
I20260925 03:01:22.262522 4029 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea9bc3b297404de68963d3dd25cd9b9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9bc3b297404de68963d3dd25cd9b9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 42135 } }
I20260925 03:01:22.262717 4029 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea9bc3b297404de68963d3dd25cd9b9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:01:22.262907 4029 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea9bc3b297404de68963d3dd25cd9b9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:01:22.263080 4029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea9bc3b297404de68963d3dd25cd9b9a [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:22.265290 4029 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea9bc3b297404de68963d3dd25cd9b9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9bc3b297404de68963d3dd25cd9b9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 42135 } }
I20260925 03:01:22.265787 4029 leader_election.cc:302] T 00000000000000000000000000000000 P ea9bc3b297404de68963d3dd25cd9b9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea9bc3b297404de68963d3dd25cd9b9a; no voters:
I20260925 03:01:22.266330 4029 leader_election.cc:288] T 00000000000000000000000000000000 P ea9bc3b297404de68963d3dd25cd9b9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:01:22.266419 4034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea9bc3b297404de68963d3dd25cd9b9a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:01:22.267570 4034 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea9bc3b297404de68963d3dd25cd9b9a [term 1 LEADER]: Becoming Leader. State: Replica: ea9bc3b297404de68963d3dd25cd9b9a, State: Running, Role: LEADER
I20260925 03:01:22.268151 4034 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea9bc3b297404de68963d3dd25cd9b9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9bc3b297404de68963d3dd25cd9b9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 42135 } }
I20260925 03:01:22.268491 4029 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea9bc3b297404de68963d3dd25cd9b9a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:01:22.271377 4036 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9bc3b297404de68963d3dd25cd9b9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea9bc3b297404de68963d3dd25cd9b9a. Latest consensus state: current_term: 1 leader_uuid: "ea9bc3b297404de68963d3dd25cd9b9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9bc3b297404de68963d3dd25cd9b9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 42135 } } }
I20260925 03:01:22.271991 4035 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9bc3b297404de68963d3dd25cd9b9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea9bc3b297404de68963d3dd25cd9b9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9bc3b297404de68963d3dd25cd9b9a" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 42135 } } }
I20260925 03:01:22.272269 4036 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9bc3b297404de68963d3dd25cd9b9a [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:22.272269 4035 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9bc3b297404de68963d3dd25cd9b9a [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:22.275780 4042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:01:22.280558 4042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:01:22.288897 4042 catalog_manager.cc:1347] Generated new cluster ID: 9efb3c31c0084e91b4b4b91bd3006f38
I20260925 03:01:22.289009 4042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:01:22.302205 4042 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:01:22.303493 4042 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:01:22.309335 4042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea9bc3b297404de68963d3dd25cd9b9a: Generated new TSK 0
I20260925 03:01:22.310074 4042 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
I20260925 03:01:22.383994 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:42135
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:01:22.701587 4055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:22.701851 4055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:22.701948 4055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:22.708973 4055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:22.709081 4055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:22.709184 4055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 03:01:22.718055 4055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:42135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:22.719306 4055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:22.720709 4055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:22.733063 4060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:22.733119 4061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:22.733150 4063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:22.734148 4055 server_base.cc:1034] running on GCE node
I20260925 03:01:22.734717 4055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:22.735671 4055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:22.736892 4055 hybrid_clock.cc:648] HybridClock initialized: now 1790305282736851 us; error 49 us; skew 500 ppm
I20260925 03:01:22.737187 4055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:22 dist-test-slave-8zvk krb5kdc[3949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305282, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:22.740095 4055 init.cc:377] Logged in from keytab as kudu/127.18.196.193@KRBTEST.COM (short username kudu)
I20260925 03:01:22.741480 4055 webserver.cc:460] Webserver started at http://127.18.196.193:44195/ using document root <none> and password file <none>
I20260925 03:01:22.742129 4055 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:22.742274 4055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:22.742580 4055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:22.745441 4055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "6e082dece5414fa494d7851edeb190fb"
format_stamp: "Formatted at 2026-09-25 03:01:22 on dist-test-slave-8zvk"
server_key: "3e56da18d66f0262f023d444b157f820"
server_key_iv: "3446d04e980b3901216fd54e471f493b"
server_key_version: "encryptionkey@0"
I20260925 03:01:22.746187 4055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "6e082dece5414fa494d7851edeb190fb"
format_stamp: "Formatted at 2026-09-25 03:01:22 on dist-test-slave-8zvk"
server_key: "3e56da18d66f0262f023d444b157f820"
server_key_iv: "3446d04e980b3901216fd54e471f493b"
server_key_version: "encryptionkey@0"
I20260925 03:01:22.751194 4055 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 03:01:22.754345 4070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:22.755661 4055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 03:01:22.755843 4055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "6e082dece5414fa494d7851edeb190fb"
format_stamp: "Formatted at 2026-09-25 03:01:22 on dist-test-slave-8zvk"
server_key: "3e56da18d66f0262f023d444b157f820"
server_key_iv: "3446d04e980b3901216fd54e471f493b"
server_key_version: "encryptionkey@0"
I20260925 03:01:22.756032 4055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:22.778206 4055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:22.782056 4055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:22.782331 4055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:22.783322 4055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:01:22.784567 4055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:01:22.784674 4055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:22.784814 4055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:01:22.784884 4055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:22.817984 4055 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:40503
I20260925 03:01:22.818069 4182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:40503 every 8 connection(s)
I20260925 03:01:22.819301 4055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 03:01:22.826782 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 4055
I20260925 03:01:22.827040 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 03:01:22.827694 19219 external_mini_cluster.cc:1428] Setting key 147cf032fc452848da09fe6e9b7dd20a
Sep 25 03:01:22 dist-test-slave-8zvk krb5kdc[3949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305282, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:01:22.839056 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:22.821788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:36707 (local address 127.18.196.254:42135)
0925 03:01:22.822272 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:22.822277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:22.823288 (+ 1011us) server_negotiation.cc:407] Connection header received
0925 03:01:22.825094 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:22.825100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:22.825194 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:22.825318 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:22.827022 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:22.827835 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:22.829502 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:22.829896 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:22.830035 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:22.833385 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:22.833390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:22.833393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:22.833423 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:22.835992 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:22.836739 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:22.836744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:22.836747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:22.836867 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:22.837318 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:22.837322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:22.837324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:22.837592 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:22.837751 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:22.838446 (+ 695us) server_negotiation.cc:299] Negotiation successful
0925 03:01:22.838674 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":224,"threads_started":1}
I20260925 03:01:22.840010 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:22.822399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:42135 (local address 127.18.196.193:36707)
0925 03:01:22.823126 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:22.823167 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:22.824841 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:22.825543 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:22.825551 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:22.826065 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:01:22.826803 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:22.826837 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:22.828010 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:22.828016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:22.829344 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:22.829356 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:22.830118 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:22.830127 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:22.830361 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:22.831051 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:01:22.831072 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:01:22.833161 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:01:22.836135 (+ 2974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:01:22.836145 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:01:22.836161 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:01:22.836601 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:01:22.836981 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:01:22.836987 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:01:22.836991 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:01:22.837187 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:01:22.837725 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:01:22.837733 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:01:22.838075 (+ 342us) client_negotiation.cc:769] Sending connection context
0925 03:01:22.838334 (+ 259us) client_negotiation.cc:240] Negotiation successful
0925 03:01:22.838888 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":266,"threads_started":1}
I20260925 03:01:22.841564 4183 heartbeater.cc:347] Connected to a master server at 127.18.196.254:42135
I20260925 03:01:22.841917 4183 heartbeater.cc:464] Registering TS with master...
I20260925 03:01:22.842741 4183 heartbeater.cc:511] Master 127.18.196.254:42135 requested a full tablet report, sending...
I20260925 03:01:22.845177 3994 ts_manager.cc:194] Registered new tserver with Master: 6e082dece5414fa494d7851edeb190fb (127.18.196.193:40503)
I20260925 03:01:22.847290 3994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.193@KRBTEST.COM'} at 127.18.196.193:36707
WARNING: no policy specified for kudu/127.18.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
I20260925 03:01:22.886348 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:42135
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:01:23.209260 4189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:23.209518 4189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:23.209715 4189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:23.217278 4189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:23.217432 4189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:23.217625 4189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 03:01:23.229467 4189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:42135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:23.230676 4189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:23.232003 4189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:23.245090 4194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:23.245059 4195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:23.245014 4197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:23.245855 4189 server_base.cc:1034] running on GCE node
I20260925 03:01:23.246480 4189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:23.247540 4189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:23.248761 4189 hybrid_clock.cc:648] HybridClock initialized: now 1790305283248707 us; error 42 us; skew 500 ppm
I20260925 03:01:23.249104 4189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:23 dist-test-slave-8zvk krb5kdc[3949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305283, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:23.252210 4189 init.cc:377] Logged in from keytab as kudu/127.18.196.194@KRBTEST.COM (short username kudu)
I20260925 03:01:23.253569 4189 webserver.cc:460] Webserver started at http://127.18.196.194:44529/ using document root <none> and password file <none>
I20260925 03:01:23.254243 4189 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:23.254396 4189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:23.254711 4189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:23.258433 4189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "316590ce05634deeb68e415062880ea6"
format_stamp: "Formatted at 2026-09-25 03:01:23 on dist-test-slave-8zvk"
server_key: "6e2161ff47aa30730cbf4aa5fd4491a5"
server_key_iv: "7079bbd48c7b5f417c3aff9e82d8b57b"
server_key_version: "encryptionkey@0"
I20260925 03:01:23.259598 4189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "316590ce05634deeb68e415062880ea6"
format_stamp: "Formatted at 2026-09-25 03:01:23 on dist-test-slave-8zvk"
server_key: "6e2161ff47aa30730cbf4aa5fd4491a5"
server_key_iv: "7079bbd48c7b5f417c3aff9e82d8b57b"
server_key_version: "encryptionkey@0"
I20260925 03:01:23.266609 4189 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260925 03:01:23.271756 4204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:23.273463 4189 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 03:01:23.273622 4189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "316590ce05634deeb68e415062880ea6"
format_stamp: "Formatted at 2026-09-25 03:01:23 on dist-test-slave-8zvk"
server_key: "6e2161ff47aa30730cbf4aa5fd4491a5"
server_key_iv: "7079bbd48c7b5f417c3aff9e82d8b57b"
server_key_version: "encryptionkey@0"
I20260925 03:01:23.273857 4189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:23.301430 4189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:23.305388 4189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:23.305676 4189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:23.306622 4189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:01:23.307981 4189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:01:23.308085 4189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:23.308236 4189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:01:23.308308 4189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:23.342195 4189 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:39569
I20260925 03:01:23.342295 4317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:39569 every 8 connection(s)
I20260925 03:01:23.343596 4189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 03:01:23.350617 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 4189
I20260925 03:01:23.350883 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 03:01:23.351346 19219 external_mini_cluster.cc:1428] Setting key 440b4bd56d801a592695608fd76ebb8f
Sep 25 03:01:23 dist-test-slave-8zvk krb5kdc[3949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305283, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:01:23.363655 4320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:23.346136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:44131 (local address 127.18.196.254:42135)
0925 03:01:23.346560 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:23.346565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:23.347597 (+ 1032us) server_negotiation.cc:407] Connection header received
0925 03:01:23.349342 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:23.349347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:23.349419 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:23.349523 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:23.351040 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:23.351821 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:23.353603 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:23.354036 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:23.354172 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:23.357783 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:23.357788 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:23.357791 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:23.357823 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:23.360609 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:23.361325 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:23.361333 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:23.361341 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:23.361476 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:23.361922 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:23.361932 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:23.361934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:23.362197 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:23.362365 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:23.363216 (+ 851us) server_negotiation.cc:299] Negotiation successful
0925 03:01:23.363384 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":210,"threads_started":1}
I20260925 03:01:23.365074 4321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:23.346686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:42135 (local address 127.18.196.194:44131)
0925 03:01:23.347435 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:23.347476 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:23.349075 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:23.349724 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:23.349732 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:23.350247 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:01:23.350826 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:23.350880 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:23.351988 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:23.351993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:23.353453 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:23.353466 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:23.354276 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:23.354283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:23.354515 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:23.355226 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:01:23.355253 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:01:23.357554 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:01:23.360742 (+ 3188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:01:23.360750 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:01:23.360766 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:01:23.361194 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:01:23.361592 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:01:23.361596 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:01:23.361600 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:01:23.361796 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:01:23.362328 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:01:23.362336 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:01:23.362779 (+ 443us) client_negotiation.cc:769] Sending connection context
0925 03:01:23.363105 (+ 326us) client_negotiation.cc:240] Negotiation successful
0925 03:01:23.363664 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":325,"threads_started":1}
I20260925 03:01:23.366459 4318 heartbeater.cc:347] Connected to a master server at 127.18.196.254:42135
I20260925 03:01:23.366771 4318 heartbeater.cc:464] Registering TS with master...
I20260925 03:01:23.367604 4318 heartbeater.cc:511] Master 127.18.196.254:42135 requested a full tablet report, sending...
I20260925 03:01:23.369186 3994 ts_manager.cc:194] Registered new tserver with Master: 316590ce05634deeb68e415062880ea6 (127.18.196.194:39569)
I20260925 03:01:23.370021 3994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.194@KRBTEST.COM'} at 127.18.196.194:44131
WARNING: no policy specified for kudu/127.18.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
I20260925 03:01:23.408378 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:42135
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:01:23.735208 4324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:23.735508 4324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:23.735674 4324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:23.743152 4324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:23.743289 4324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:23.743458 4324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 03:01:23.753178 4324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:42135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:23.754374 4324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:23.755724 4324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:23.768529 4332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:23.768502 4330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:23.768949 4329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:23.769847 4324 server_base.cc:1034] running on GCE node
I20260925 03:01:23.770363 4324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:23.771350 4324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:23.772544 4324 hybrid_clock.cc:648] HybridClock initialized: now 1790305283772495 us; error 50 us; skew 500 ppm
I20260925 03:01:23.772787 4324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:23 dist-test-slave-8zvk krb5kdc[3949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305283, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:23.775990 4324 init.cc:377] Logged in from keytab as kudu/127.18.196.195@KRBTEST.COM (short username kudu)
I20260925 03:01:23.777276 4324 webserver.cc:460] Webserver started at http://127.18.196.195:40963/ using document root <none> and password file <none>
I20260925 03:01:23.777949 4324 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:23.778080 4324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:23.778385 4324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:23.781332 4324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "6500eaa94f5d4168b305f05a860337f9"
format_stamp: "Formatted at 2026-09-25 03:01:23 on dist-test-slave-8zvk"
server_key: "5d60073fcbb357d48226ea4b16d12f2c"
server_key_iv: "73c89faeb16b2c5d6110717dadd0d9c6"
server_key_version: "encryptionkey@0"
I20260925 03:01:23.782126 4324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "6500eaa94f5d4168b305f05a860337f9"
format_stamp: "Formatted at 2026-09-25 03:01:23 on dist-test-slave-8zvk"
server_key: "5d60073fcbb357d48226ea4b16d12f2c"
server_key_iv: "73c89faeb16b2c5d6110717dadd0d9c6"
server_key_version: "encryptionkey@0"
I20260925 03:01:23.787000 4324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260925 03:01:23.790107 4339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:23.791342 4324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 03:01:23.791501 4324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "6500eaa94f5d4168b305f05a860337f9"
format_stamp: "Formatted at 2026-09-25 03:01:23 on dist-test-slave-8zvk"
server_key: "5d60073fcbb357d48226ea4b16d12f2c"
server_key_iv: "73c89faeb16b2c5d6110717dadd0d9c6"
server_key_version: "encryptionkey@0"
I20260925 03:01:23.791677 4324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:23.821985 4324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:23.825820 4324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:23.826112 4324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:23.827099 4324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:01:23.828332 4324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:01:23.828404 4324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:23.828540 4324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:01:23.828625 4324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:23.851044 4183 heartbeater.cc:503] Master 127.18.196.254:42135 was elected leader, sending a full tablet report...
I20260925 03:01:23.861088 4324 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:38121
I20260925 03:01:23.861167 4451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:38121 every 8 connection(s)
I20260925 03:01:23.862366 4324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 03:01:23.863823 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 4324
I20260925 03:01:23.864044 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 03:01:23.864645 19219 external_mini_cluster.cc:1428] Setting key 774a2d15e1997dfea80cc0613cfb0506
Sep 25 03:01:23 dist-test-slave-8zvk krb5kdc[3949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305283, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:01:23.882221 4454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:23.864919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:60847 (local address 127.18.196.254:42135)
0925 03:01:23.865514 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:23.865520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:23.866557 (+ 1037us) server_negotiation.cc:407] Connection header received
0925 03:01:23.868690 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:23.868695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:23.868796 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:23.868911 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:23.870383 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:23.871236 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:23.872724 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:23.873123 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:23.873250 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:23.876564 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:23.876569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:23.876573 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:23.876604 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:23.879447 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:23.880347 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:23.880353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:23.880362 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:23.880481 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:23.880852 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:23.880855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:23.880857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:23.881098 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:23.881215 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:23.881813 (+ 598us) server_negotiation.cc:299] Negotiation successful
0925 03:01:23.881971 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":340,"threads_started":1}
I20260925 03:01:23.883188 4455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:23.865585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:42135 (local address 127.18.196.195:60847)
0925 03:01:23.866415 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:23.866463 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:23.868384 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:23.869137 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:23.869145 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:23.869680 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:01:23.870213 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:23.870230 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:23.871406 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:23.871416 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:23.872578 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:23.872590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:23.873343 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:23.873350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:23.873574 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:23.874220 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:01:23.874241 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:01:23.876360 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:01:23.879578 (+ 3218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:01:23.879584 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:01:23.879594 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:01:23.880188 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:01:23.880574 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:01:23.880577 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:01:23.880580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:01:23.880758 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:01:23.881205 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:01:23.881210 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:01:23.881502 (+ 292us) client_negotiation.cc:769] Sending connection context
0925 03:01:23.881721 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 03:01:23.882073 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":356,"threads_started":1}
I20260925 03:01:23.884439 4452 heartbeater.cc:347] Connected to a master server at 127.18.196.254:42135
I20260925 03:01:23.884734 4452 heartbeater.cc:464] Registering TS with master...
I20260925 03:01:23.885414 4452 heartbeater.cc:511] Master 127.18.196.254:42135 requested a full tablet report, sending...
I20260925 03:01:23.886718 3994 ts_manager.cc:194] Registered new tserver with Master: 6500eaa94f5d4168b305f05a860337f9 (127.18.196.195:38121)
I20260925 03:01:23.887511 3994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.195@KRBTEST.COM'} at 127.18.196.195:60847
I20260925 03:01:23.892079 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 03:01:23.908181 4454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:23.896463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35216 (local address 127.18.196.254:42135)
0925 03:01:23.896611 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:23.896615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:23.897074 (+ 459us) server_negotiation.cc:407] Connection header received
0925 03:01:23.897197 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:23.897201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:23.897267 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:23.897345 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:23.898742 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:23.899570 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:23.900797 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:23.901167 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:23.901256 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:23.903521 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:23.903525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:23.903528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:23.903553 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:23.905788 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:23.906423 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:23.906427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:23.906429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:23.906534 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:23.906991 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:23.906994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:23.906996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:23.907288 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:23.907465 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:23.907804 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 03:01:23.907963 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 03:01:23.911777 3994 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 03:01:23.918394 3994 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 03:01:23.942795 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:23.929587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:38121 (local address 127.0.0.1:50990)
0925 03:01:23.930507 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:23.930548 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:23.930726 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:23.931986 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:23.931997 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:23.932030 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:23.932488 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:23.932508 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:23.935447 (+ 2939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:23.935455 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:23.938693 (+ 3238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:23.938715 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:23.941261 (+ 2546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:23.941270 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:23.941582 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:23.941655 (+ 73us) client_negotiation.cc:769] Sending connection context
0925 03:01:23.941809 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 03:01:23.942132 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":1,"thread_start_us":519,"threads_started":1}
I20260925 03:01:23.943608 4468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:23.930846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.18.196.194:39569)
0925 03:01:23.932103 (+ 1257us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:23.932115 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:23.932147 (+ 32us) server_negotiation.cc:407] Connection header received
0925 03:01:23.932262 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:23.932272 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:23.932520 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:23.932728 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:23.933791 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:23.935766 (+ 1975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:23.939190 (+ 3424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:23.941156 (+ 1966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:23.941359 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:23.941588 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:23.943095 (+ 1507us) server_negotiation.cc:299] Negotiation successful
0925 03:01:23.943269 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1052,"thread_start_us":338,"threads_started":1}
I20260925 03:01:23.944231 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:23.929946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50990 (local address 127.18.196.195:38121)
0925 03:01:23.931382 (+ 1436us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:23.931392 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:23.931425 (+ 33us) server_negotiation.cc:407] Connection header received
0925 03:01:23.931501 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:23.931506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:23.931730 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:23.932770 (+ 1040us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:23.933100 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:23.935231 (+ 2131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:23.938990 (+ 3759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:23.940963 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:23.943497 (+ 2534us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:23.943675 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:23.943743 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 03:01:23.943898 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1262,"thread_start_us":1095,"threads_started":1}
I20260925 03:01:23.944423 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:23.930481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:39569 (local address 127.0.0.1:36200)
0925 03:01:23.931587 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:23.931605 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:23.931715 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:23.933030 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:23.933037 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:23.933071 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:23.933570 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:23.933585 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:23.935991 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:23.935999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:23.938953 (+ 2954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:23.938974 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:23.941793 (+ 2819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:23.941803 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:23.942857 (+ 1054us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:23.942915 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 03:01:23.943879 (+ 964us) client_negotiation.cc:240] Negotiation successful
0925 03:01:23.944019 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979,"mutex_wait_us":746,"thread_start_us":874,"threads_started":1}
I20260925 03:01:23.948050 4252 tablet_service.cc:1467] Processing CreateTablet for tablet f6ebeee87b4b470eb50457ca24843cd9 (DEFAULT_TABLE table=test-table [id=da16d11a28b845a489c1eab87b245ba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 03:01:23.948102 4387 tablet_service.cc:1467] Processing CreateTablet for tablet f6ebeee87b4b470eb50457ca24843cd9 (DEFAULT_TABLE table=test-table [id=da16d11a28b845a489c1eab87b245ba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 03:01:23.949404 4252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ebeee87b4b470eb50457ca24843cd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:23.949465 4387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ebeee87b4b470eb50457ca24843cd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:23.951969 4469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:23.931715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:40503 (local address 127.0.0.1:40948)
0925 03:01:23.942517 (+ 10802us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:23.942539 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:23.942650 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:23.944943 (+ 2293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:23.944947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:23.944973 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:23.945316 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:23.945331 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:23.947290 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:23.947295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:23.949307 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:23.949323 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:23.951206 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:23.951212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:23.951370 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:23.951402 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 03:01:23.951504 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 03:01:23.951636 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10675,"thread_start_us":10508,"threads_started":1}
I20260925 03:01:23.952054 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:23.932089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40948 (local address 127.18.196.193:40503)
0925 03:01:23.944376 (+ 12287us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:23.944385 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:23.944414 (+ 29us) server_negotiation.cc:407] Connection header received
0925 03:01:23.944512 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:23.944520 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:23.944715 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:23.944957 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:23.945529 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:23.947141 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:23.949507 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:23.950976 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:23.951171 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:23.951333 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:23.951566 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 03:01:23.951718 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12088,"thread_start_us":11915,"threads_started":1}
I20260925 03:01:23.955302 4118 tablet_service.cc:1467] Processing CreateTablet for tablet f6ebeee87b4b470eb50457ca24843cd9 (DEFAULT_TABLE table=test-table [id=da16d11a28b845a489c1eab87b245ba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 03:01:23.956574 4118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ebeee87b4b470eb50457ca24843cd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:23.961891 4471 tablet_bootstrap.cc:492] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9: Bootstrap starting.
I20260925 03:01:23.965437 4471 tablet_bootstrap.cc:654] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:23.966426 4471 log.cc:824] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:23.967720 4472 tablet_bootstrap.cc:492] T f6ebeee87b4b470eb50457ca24843cd9 P 316590ce05634deeb68e415062880ea6: Bootstrap starting.
I20260925 03:01:23.970912 4471 tablet_bootstrap.cc:492] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9: No bootstrap required, opened a new log
I20260925 03:01:23.971244 4471 ts_tablet_manager.cc:1397] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260925 03:01:23.971719 4472 tablet_bootstrap.cc:654] T f6ebeee87b4b470eb50457ca24843cd9 P 316590ce05634deeb68e415062880ea6: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:23.972208 4474 tablet_bootstrap.cc:492] T f6ebeee87b4b470eb50457ca24843cd9 P 6e082dece5414fa494d7851edeb190fb: Bootstrap starting.
I20260925 03:01:23.972920 4472 log.cc:824] T f6ebeee87b4b470eb50457ca24843cd9 P 316590ce05634deeb68e415062880ea6: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:23.975214 4474 tablet_bootstrap.cc:654] T f6ebeee87b4b470eb50457ca24843cd9 P 6e082dece5414fa494d7851edeb190fb: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:23.975965 4472 tablet_bootstrap.cc:492] T f6ebeee87b4b470eb50457ca24843cd9 P 316590ce05634deeb68e415062880ea6: No bootstrap required, opened a new log
I20260925 03:01:23.976214 4472 ts_tablet_manager.cc:1397] T f6ebeee87b4b470eb50457ca24843cd9 P 316590ce05634deeb68e415062880ea6: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260925 03:01:23.976248 4474 log.cc:824] T f6ebeee87b4b470eb50457ca24843cd9 P 6e082dece5414fa494d7851edeb190fb: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:23.978515 4474 tablet_bootstrap.cc:492] T f6ebeee87b4b470eb50457ca24843cd9 P 6e082dece5414fa494d7851edeb190fb: No bootstrap required, opened a new log
I20260925 03:01:23.978745 4474 ts_tablet_manager.cc:1397] T f6ebeee87b4b470eb50457ca24843cd9 P 6e082dece5414fa494d7851edeb190fb: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 03:01:23.981624 4471 raft_consensus.cc:348] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6500eaa94f5d4168b305f05a860337f9" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38121 } } peers { permanent_uuid: "316590ce05634deeb68e415062880ea6" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39569 } } peers { permanent_uuid: "6e082dece5414fa494d7851edeb190fb" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 40503 } }
I20260925 03:01:23.982074 4471 raft_consensus.cc:374] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:23.982172 4471 raft_consensus.cc:729] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6500eaa94f5d4168b305f05a860337f9, State: Initialized, Role: FOLLOWER
I20260925 03:01:23.982752 4471 consensus_queue.cc:260] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6500eaa94f5d4168b305f05a860337f9" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38121 } } peers { permanent_uuid: "316590ce05634deeb68e415062880ea6" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39569 } } peers { permanent_uuid: "6e082dece5414fa494d7851edeb190fb" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 40503 } }
I20260925 03:01:23.983888 4452 heartbeater.cc:503] Master 127.18.196.254:42135 was elected leader, sending a full tablet report...
I20260925 03:01:23.984488 4471 ts_tablet_manager.cc:1428] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260925 03:01:23.986814 4472 raft_consensus.cc:348] T f6ebeee87b4b470eb50457ca24843cd9 P 316590ce05634deeb68e415062880ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6500eaa94f5d4168b305f05a860337f9" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38121 } } peers { permanent_uuid: "316590ce05634deeb68e415062880ea6" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39569 } } peers { permanent_uuid: "6e082dece5414fa494d7851edeb190fb" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 40503 } }
I20260925 03:01:23.987215 4472 raft_consensus.cc:374] T f6ebeee87b4b470eb50457ca24843cd9 P 316590ce05634deeb68e415062880ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:23.987371 4472 raft_consensus.cc:729] T f6ebeee87b4b470eb50457ca24843cd9 P 316590ce05634deeb68e415062880ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316590ce05634deeb68e415062880ea6, State: Initialized, Role: FOLLOWER
I20260925 03:01:23.987623 4474 raft_consensus.cc:348] T f6ebeee87b4b470eb50457ca24843cd9 P 6e082dece5414fa494d7851edeb190fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6500eaa94f5d4168b305f05a860337f9" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38121 } } peers { permanent_uuid: "316590ce05634deeb68e415062880ea6" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39569 } } peers { permanent_uuid: "6e082dece5414fa494d7851edeb190fb" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 40503 } }
I20260925 03:01:23.988036 4474 raft_consensus.cc:374] T f6ebeee87b4b470eb50457ca24843cd9 P 6e082dece5414fa494d7851edeb190fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:23.988160 4474 raft_consensus.cc:729] T f6ebeee87b4b470eb50457ca24843cd9 P 6e082dece5414fa494d7851edeb190fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e082dece5414fa494d7851edeb190fb, State: Initialized, Role: FOLLOWER
I20260925 03:01:23.988059 4472 consensus_queue.cc:260] T f6ebeee87b4b470eb50457ca24843cd9 P 316590ce05634deeb68e415062880ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6500eaa94f5d4168b305f05a860337f9" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38121 } } peers { permanent_uuid: "316590ce05634deeb68e415062880ea6" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39569 } } peers { permanent_uuid: "6e082dece5414fa494d7851edeb190fb" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 40503 } }
I20260925 03:01:23.988724 4474 consensus_queue.cc:260] T f6ebeee87b4b470eb50457ca24843cd9 P 6e082dece5414fa494d7851edeb190fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6500eaa94f5d4168b305f05a860337f9" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38121 } } peers { permanent_uuid: "316590ce05634deeb68e415062880ea6" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39569 } } peers { permanent_uuid: "6e082dece5414fa494d7851edeb190fb" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 40503 } }
I20260925 03:01:23.989797 4472 ts_tablet_manager.cc:1428] T f6ebeee87b4b470eb50457ca24843cd9 P 316590ce05634deeb68e415062880ea6: Time spent starting tablet: real 0.013s user 0.005s sys 0.007s
I20260925 03:01:23.989876 4318 heartbeater.cc:503] Master 127.18.196.254:42135 was elected leader, sending a full tablet report...
I20260925 03:01:23.990414 4474 ts_tablet_manager.cc:1428] T f6ebeee87b4b470eb50457ca24843cd9 P 6e082dece5414fa494d7851edeb190fb: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260925 03:01:24.061221 4477 raft_consensus.cc:482] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:01:24.061502 4477 raft_consensus.cc:504] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6500eaa94f5d4168b305f05a860337f9" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38121 } } peers { permanent_uuid: "316590ce05634deeb68e415062880ea6" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39569 } } peers { permanent_uuid: "6e082dece5414fa494d7851edeb190fb" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 40503 } }
I20260925 03:01:24.063273 4477 leader_election.cc:288] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316590ce05634deeb68e415062880ea6 (127.18.196.194:39569), 6e082dece5414fa494d7851edeb190fb (127.18.196.193:40503)
I20260925 03:01:24.069142 4455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:24.063644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:39569 (local address 127.18.196.195:40987)
0925 03:01:24.063830 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:24.063847 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:24.063978 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:24.064333 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:24.064338 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:24.064364 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:24.064574 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:24.064583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:24.065779 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:24.065782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:24.067420 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:24.067435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:24.068527 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:24.068530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:24.068654 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:24.068677 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 03:01:24.068745 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 03:01:24.068951 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 03:01:24.069159 4468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:24.063665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:40987 (local address 127.18.196.194:39569)
0925 03:01:24.063843 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:24.063847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:24.063947 (+ 100us) server_negotiation.cc:407] Connection header received
0925 03:01:24.064137 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:24.064144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:24.064218 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:24.064313 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:24.064710 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:24.065657 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:24.067565 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:24.068398 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:24.068538 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:24.068609 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:24.068794 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 03:01:24.068914 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 03:01:24.070138 4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ebeee87b4b470eb50457ca24843cd9" candidate_uuid: "6500eaa94f5d4168b305f05a860337f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316590ce05634deeb68e415062880ea6" is_pre_election: true
I20260925 03:01:24.070349 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:24.064157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:46897 (local address 127.18.196.193:40503)
0925 03:01:24.064402 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:24.064411 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:24.065005 (+ 594us) server_negotiation.cc:407] Connection header received
0925 03:01:24.065105 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:24.065112 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:24.065183 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:24.065273 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:24.065755 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:24.066460 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:24.068336 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:24.069538 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:24.069694 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:24.069753 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:24.070110 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 03:01:24.070195 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 03:01:24.070537 4480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:24.064001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:40503 (local address 127.18.196.195:46897)
0925 03:01:24.064903 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:24.064916 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:24.065007 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:24.065293 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:24.065295 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:24.065312 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:24.065587 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:24.065594 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:24.066611 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:24.066616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:24.068193 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:24.068204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:24.069736 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:24.069744 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:24.069931 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:24.069962 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 03:01:24.070076 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 03:01:24.070206 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":623,"threads_started":1}
I20260925 03:01:24.070665 4273 raft_consensus.cc:2454] T f6ebeee87b4b470eb50457ca24843cd9 P 316590ce05634deeb68e415062880ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6500eaa94f5d4168b305f05a860337f9 in term 0.
I20260925 03:01:24.071310 4343 leader_election.cc:302] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316590ce05634deeb68e415062880ea6, 6500eaa94f5d4168b305f05a860337f9; no voters:
W20260925 03:01:24.071617 4184 tablet.cc:2369] T f6ebeee87b4b470eb50457ca24843cd9 P 6e082dece5414fa494d7851edeb190fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 03:01:24.071695 4477 raft_consensus.cc:2781] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:01:24.071853 4477 raft_consensus.cc:482] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:01:24.071964 4477 raft_consensus.cc:3037] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:24.071960 4138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ebeee87b4b470eb50457ca24843cd9" candidate_uuid: "6500eaa94f5d4168b305f05a860337f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e082dece5414fa494d7851edeb190fb" is_pre_election: true
I20260925 03:01:24.072471 4138 raft_consensus.cc:2454] T f6ebeee87b4b470eb50457ca24843cd9 P 6e082dece5414fa494d7851edeb190fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6500eaa94f5d4168b305f05a860337f9 in term 0.
I20260925 03:01:24.074325 4477 raft_consensus.cc:504] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6500eaa94f5d4168b305f05a860337f9" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38121 } } peers { permanent_uuid: "316590ce05634deeb68e415062880ea6" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39569 } } peers { permanent_uuid: "6e082dece5414fa494d7851edeb190fb" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 40503 } }
I20260925 03:01:24.074959 4477 leader_election.cc:288] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9 [CANDIDATE]: Term 1 election: Requested vote from peers 316590ce05634deeb68e415062880ea6 (127.18.196.194:39569), 6e082dece5414fa494d7851edeb190fb (127.18.196.193:40503)
I20260925 03:01:24.075354 4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ebeee87b4b470eb50457ca24843cd9" candidate_uuid: "6500eaa94f5d4168b305f05a860337f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316590ce05634deeb68e415062880ea6"
I20260925 03:01:24.075426 4138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ebeee87b4b470eb50457ca24843cd9" candidate_uuid: "6500eaa94f5d4168b305f05a860337f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e082dece5414fa494d7851edeb190fb"
I20260925 03:01:24.075579 4273 raft_consensus.cc:3037] T f6ebeee87b4b470eb50457ca24843cd9 P 316590ce05634deeb68e415062880ea6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:24.075595 4138 raft_consensus.cc:3037] T f6ebeee87b4b470eb50457ca24843cd9 P 6e082dece5414fa494d7851edeb190fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:24.077770 4138 raft_consensus.cc:2454] T f6ebeee87b4b470eb50457ca24843cd9 P 6e082dece5414fa494d7851edeb190fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6500eaa94f5d4168b305f05a860337f9 in term 1.
I20260925 03:01:24.077896 4273 raft_consensus.cc:2454] T f6ebeee87b4b470eb50457ca24843cd9 P 316590ce05634deeb68e415062880ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6500eaa94f5d4168b305f05a860337f9 in term 1.
I20260925 03:01:24.078387 4343 leader_election.cc:302] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316590ce05634deeb68e415062880ea6, 6500eaa94f5d4168b305f05a860337f9; no voters:
I20260925 03:01:24.078624 4477 raft_consensus.cc:2781] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:01:24.079092 4477 raft_consensus.cc:686] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6500eaa94f5d4168b305f05a860337f9, State: Running, Role: LEADER
I20260925 03:01:24.079591 4477 consensus_queue.cc:237] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6500eaa94f5d4168b305f05a860337f9" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38121 } } peers { permanent_uuid: "316590ce05634deeb68e415062880ea6" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39569 } } peers { permanent_uuid: "6e082dece5414fa494d7851edeb190fb" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 40503 } }
I20260925 03:01:24.083669 3994 catalog_manager.cc:5495] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6500eaa94f5d4168b305f05a860337f9 (127.18.196.195). New cstate: current_term: 1 leader_uuid: "6500eaa94f5d4168b305f05a860337f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6500eaa94f5d4168b305f05a860337f9" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 38121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "316590ce05634deeb68e415062880ea6" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e082dece5414fa494d7851edeb190fb" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 40503 } health_report { overall_health: UNKNOWN } } }
W20260925 03:01:24.095566 4319 tablet.cc:2369] T f6ebeee87b4b470eb50457ca24843cd9 P 316590ce05634deeb68e415062880ea6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 03:01:24.104311 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:24.099038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50992 (local address 127.18.196.195:38121)
0925 03:01:24.099228 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:24.099234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:24.099306 (+ 72us) server_negotiation.cc:407] Connection header received
0925 03:01:24.099551 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:24.099555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:24.099623 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:24.099765 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:01:24.100164 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:24.100898 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:24.102306 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:24.102699 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:24.102832 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:24.103341 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:01:24.103771 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:01:24.103933 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:24.103988 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 03:01:24.104063 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 03:01:24.111877 4273 raft_consensus.cc:1261] T f6ebeee87b4b470eb50457ca24843cd9 P 316590ce05634deeb68e415062880ea6 [term 1 FOLLOWER]: Refusing update from remote peer 6500eaa94f5d4168b305f05a860337f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:01:24.111923 4138 raft_consensus.cc:1261] T f6ebeee87b4b470eb50457ca24843cd9 P 6e082dece5414fa494d7851edeb190fb [term 1 FOLLOWER]: Refusing update from remote peer 6500eaa94f5d4168b305f05a860337f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:01:24.112803 4481 consensus_queue.cc:1035] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316590ce05634deeb68e415062880ea6" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:01:24.113150 4477 consensus_queue.cc:1035] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e082dece5414fa494d7851edeb190fb" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 40503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 03:01:24.114066 4453 tablet.cc:2369] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 03:01:24.137586 3992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 03:01:24.137908 3992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 03:01:24.141551 3992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea9bc3b297404de68963d3dd25cd9b9a: Sending DeleteTablet for 3 replicas of tablet f6ebeee87b4b470eb50457ca24843cd9
I20260925 03:01:24.142589 4387 tablet_service.cc:1514] Processing DeleteTablet for tablet f6ebeee87b4b470eb50457ca24843cd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 03:01:24 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:50990
I20260925 03:01:24.142668 4118 tablet_service.cc:1514] Processing DeleteTablet for tablet f6ebeee87b4b470eb50457ca24843cd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 03:01:24 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:40948
I20260925 03:01:24.142962 4252 tablet_service.cc:1514] Processing DeleteTablet for tablet f6ebeee87b4b470eb50457ca24843cd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 03:01:24 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:36200
I20260925 03:01:24.144156 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 4055
I20260925 03:01:24.144546 4494 tablet_replica.cc:330] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9: stopping tablet replica
I20260925 03:01:24.144953 4494 raft_consensus.cc:2229] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 03:01:24.145542 4494 raft_consensus.cc:2258] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:01:24.147475 4494 ts_tablet_manager.cc:1905] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 03:01:24.154910 4494 ts_tablet_manager.cc:1918] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 03:01:24.155076 4494 log.cc:1196] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/f6ebeee87b4b470eb50457ca24843cd9
I20260925 03:01:24.155606 4494 ts_tablet_manager.cc:1939] T f6ebeee87b4b470eb50457ca24843cd9 P 6500eaa94f5d4168b305f05a860337f9: Deleting consensus metadata
I20260925 03:01:24.157231 3979 catalog_manager.cc:4841] TS 6500eaa94f5d4168b305f05a860337f9 (127.18.196.195:38121): tablet f6ebeee87b4b470eb50457ca24843cd9 (table test-table [id=da16d11a28b845a489c1eab87b245ba6]) successfully deleted
I20260925 03:01:24.158715 4496 tablet_replica.cc:330] T f6ebeee87b4b470eb50457ca24843cd9 P 316590ce05634deeb68e415062880ea6: stopping tablet replica
I20260925 03:01:24.159132 4496 raft_consensus.cc:2229] T f6ebeee87b4b470eb50457ca24843cd9 P 316590ce05634deeb68e415062880ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:01:24.159593 4496 raft_consensus.cc:2258] T f6ebeee87b4b470eb50457ca24843cd9 P 316590ce05634deeb68e415062880ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:01:24.160279 4495 tablet_replica.cc:330] T f6ebeee87b4b470eb50457ca24843cd9 P 6e082dece5414fa494d7851edeb190fb: stopping tablet replica
I20260925 03:01:24.160691 4495 raft_consensus.cc:2229] T f6ebeee87b4b470eb50457ca24843cd9 P 6e082dece5414fa494d7851edeb190fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:01:24.161188 4495 raft_consensus.cc:2258] T f6ebeee87b4b470eb50457ca24843cd9 P 6e082dece5414fa494d7851edeb190fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:01:24.162457 4496 ts_tablet_manager.cc:1905] T f6ebeee87b4b470eb50457ca24843cd9 P 316590ce05634deeb68e415062880ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 03:01:24.163105 4495 ts_tablet_manager.cc:1905] T f6ebeee87b4b470eb50457ca24843cd9 P 6e082dece5414fa494d7851edeb190fb: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 03:01:24 dist-test-slave-8zvk krb5kdc[3949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.193@KRBTEST.COM
I20260925 03:01:24.169952 4496 ts_tablet_manager.cc:1918] T f6ebeee87b4b470eb50457ca24843cd9 P 316590ce05634deeb68e415062880ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 03:01:24.170187 4496 log.cc:1196] T f6ebeee87b4b470eb50457ca24843cd9 P 316590ce05634deeb68e415062880ea6: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/f6ebeee87b4b470eb50457ca24843cd9
I20260925 03:01:24.170392 4495 ts_tablet_manager.cc:1918] T f6ebeee87b4b470eb50457ca24843cd9 P 6e082dece5414fa494d7851edeb190fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 03:01:24.170558 4495 log.cc:1196] T f6ebeee87b4b470eb50457ca24843cd9 P 6e082dece5414fa494d7851edeb190fb: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304987644417-19219-0/minicluster-data/ts-0/wal/wals/f6ebeee87b4b470eb50457ca24843cd9
I20260925 03:01:24.170974 4496 ts_tablet_manager.cc:1939] T f6ebeee87b4b470eb50457ca24843cd9 P 316590ce05634deeb68e415062880ea6: Deleting consensus metadata
I20260925 03:01:24.171142 4495 ts_tablet_manager.cc:1939] T f6ebeee87b4b470eb50457ca24843cd9 P 6e082dece5414fa494d7851edeb190fb: Deleting consensus metadata
I20260925 03:01:24.172883 3981 catalog_manager.cc:4841] TS 6e082dece5414fa494d7851edeb190fb (127.18.196.193:40503): tablet f6ebeee87b4b470eb50457ca24843cd9 (table test-table [id=da16d11a28b845a489c1eab87b245ba6]) successfully deleted
I20260925 03:01:24.173171 3981 catalog_manager.cc:4841] TS 316590ce05634deeb68e415062880ea6 (127.18.196.194:39569): tablet f6ebeee87b4b470eb50457ca24843cd9 (table test-table [id=da16d11a28b845a489c1eab87b245ba6]) successfully deleted
I20260925 03:01:24.176961 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:24.146078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40958 (local address 127.18.196.193:40503)
0925 03:01:24.146291 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:24.146300 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:24.158075 (+ 11775us) server_negotiation.cc:407] Connection header received
0925 03:01:24.158265 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:24.158271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:24.158386 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:24.158497 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:24.160311 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:24.161504 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:24.163867 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:24.164518 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:24.164614 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:24.170237 (+ 5623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:24.170252 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:24.170270 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:24.170319 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:24.173946 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:24.174734 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:24.174749 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:24.174751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:24.174970 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:24.175376 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:24.175381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:24.175385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:24.175751 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:24.176072 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:24.176396 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 03:01:24.176608 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 03:01:24.177562 4178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:24.278432 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 4055
I20260925 03:01:24.298055 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 4189
Sep 25 03:01:24 dist-test-slave-8zvk krb5kdc[3949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.194@KRBTEST.COM
I20260925 03:01:24.313663 4468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:24.299270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36210 (local address 127.18.196.194:39569)
0925 03:01:24.299472 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:24.299477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:24.299492 (+ 15us) server_negotiation.cc:407] Connection header received
0925 03:01:24.299544 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:24.299548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:24.299607 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:24.299722 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:24.301013 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:24.301770 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:24.303124 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:24.303521 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:24.303655 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:24.307829 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:24.307843 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:24.307857 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:24.307891 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:24.310676 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:24.311616 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:24.311623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:24.311625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:24.311756 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:24.312150 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:24.312155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:24.312157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:24.312503 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:24.312826 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:24.313116 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 03:01:24.313339 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 03:01:24.314320 4313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:24.405405 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 4189
I20260925 03:01:24.426575 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 4324
Sep 25 03:01:24 dist-test-slave-8zvk krb5kdc[3949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.195@KRBTEST.COM
I20260925 03:01:24.442544 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:24.427466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50994 (local address 127.18.196.195:38121)
0925 03:01:24.427695 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:24.427701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:24.427725 (+ 24us) server_negotiation.cc:407] Connection header received
0925 03:01:24.427842 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:24.427848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:24.427948 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:24.428057 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:24.429341 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:24.430194 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:24.431562 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:24.432086 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:24.432231 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:24.436539 (+ 4308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:24.436552 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:24.436568 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:24.436601 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:24.439700 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:24.440408 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:24.440415 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:24.440418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:24.440555 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:24.440955 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:24.440961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:24.440964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:24.441326 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:24.441678 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:24.441951 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 03:01:24.442170 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 03:01:24.443302 4447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:24.567490 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 4324
I20260925 03:01:24.587980 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 3962
I20260925 03:01:24.589174 4024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:24.700764 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 3962
2026-09-25T03:01:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4220 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T03:01:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:01:24Z Disabled control of system clock
I20260925 03:01:24.778283 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:33285
--webserver_interface=127.18.196.254
--webserver_port=0
--builtin_ntp_servers=127.18.196.212:33063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:33285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 03:01:25.105021 4505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:25.105290 4505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:25.105456 4505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:25.112588 4505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:01:25.112685 4505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:25.112778 4505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:25.112884 4505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:01:25.112965 4505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:01:25.121969 4505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:33063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:33285
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:33285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:25.123225 4505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:25.125007 4505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:25.136339 4511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:25.136464 4510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:25.136443 4513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:25.137367 4505 server_base.cc:1034] running on GCE node
I20260925 03:01:25.138137 4505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:25.139552 4505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:25.140751 4505 hybrid_clock.cc:648] HybridClock initialized: now 1790305285140708 us; error 32 us; skew 500 ppm
I20260925 03:01:25.141101 4505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:01:25.143327 4505 webserver.cc:460] Webserver started at http://127.18.196.254:46329/ using document root <none> and password file <none>
I20260925 03:01:25.143986 4505 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:25.144099 4505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:25.144428 4505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:25.147282 4505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "087d87d20c7945cbac9390a0e06bc0f6"
format_stamp: "Formatted at 2026-09-25 03:01:25 on dist-test-slave-8zvk"
server_key: "df7d1244c06588d8a66f94b8568ca1b7"
server_key_iv: "48bb971b27b966e67a2ab3eb75c219dd"
server_key_version: "encryptionkey@0"
I20260925 03:01:25.148232 4505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "087d87d20c7945cbac9390a0e06bc0f6"
format_stamp: "Formatted at 2026-09-25 03:01:25 on dist-test-slave-8zvk"
server_key: "df7d1244c06588d8a66f94b8568ca1b7"
server_key_iv: "48bb971b27b966e67a2ab3eb75c219dd"
server_key_version: "encryptionkey@0"
I20260925 03:01:25.153190 4505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 03:01:25.156481 4519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:25.157764 4505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 03:01:25.157963 4505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "087d87d20c7945cbac9390a0e06bc0f6"
format_stamp: "Formatted at 2026-09-25 03:01:25 on dist-test-slave-8zvk"
server_key: "df7d1244c06588d8a66f94b8568ca1b7"
server_key_iv: "48bb971b27b966e67a2ab3eb75c219dd"
server_key_version: "encryptionkey@0"
I20260925 03:01:25.158182 4505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:25.177836 4505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:25.178637 4505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:25.178943 4505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:25.197579 4505 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:33285
I20260925 03:01:25.197598 4570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:33285 every 8 connection(s)
I20260925 03:01:25.198992 4505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:01:25.201224 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 4505
I20260925 03:01:25.201406 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:01:25.202008 19219 external_mini_cluster.cc:1428] Setting key f557386eea4fa2f28c45be927ca68b9d
I20260925 03:01:25.203214 4571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:25.217509 4574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:25.203883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46690 (local address 127.18.196.254:33285)
0925 03:01:25.204913 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:25.204927 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:25.204979 (+ 52us) server_negotiation.cc:407] Connection header received
0925 03:01:25.206752 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:25.206785 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:25.207376 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:25.207840 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:25.208336 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:25.210351 (+ 2015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:25.211850 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:25.212837 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:25.213028 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:25.214491 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:25.214502 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:25.214519 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:25.215262 (+ 743us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:25.215374 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:25.215400 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:25.215528 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:25.215889 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 03:01:25.216322 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":322,"threads_started":1}
I20260925 03:01:25.218462 4571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 087d87d20c7945cbac9390a0e06bc0f6: Bootstrap starting.
I20260925 03:01:25.222103 4571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 087d87d20c7945cbac9390a0e06bc0f6: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:25.223146 4571 log.cc:824] T 00000000000000000000000000000000 P 087d87d20c7945cbac9390a0e06bc0f6: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:25.225621 4571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 087d87d20c7945cbac9390a0e06bc0f6: No bootstrap required, opened a new log
I20260925 03:01:25.232596 4571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 087d87d20c7945cbac9390a0e06bc0f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087d87d20c7945cbac9390a0e06bc0f6" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33285 } }
I20260925 03:01:25.232965 4571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 087d87d20c7945cbac9390a0e06bc0f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:25.233072 4571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 087d87d20c7945cbac9390a0e06bc0f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 087d87d20c7945cbac9390a0e06bc0f6, State: Initialized, Role: FOLLOWER
I20260925 03:01:25.233594 4571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 087d87d20c7945cbac9390a0e06bc0f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087d87d20c7945cbac9390a0e06bc0f6" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33285 } }
I20260925 03:01:25.233827 4571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 087d87d20c7945cbac9390a0e06bc0f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:01:25.233968 4571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 087d87d20c7945cbac9390a0e06bc0f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:01:25.234140 4571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 087d87d20c7945cbac9390a0e06bc0f6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:25.236292 4571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 087d87d20c7945cbac9390a0e06bc0f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087d87d20c7945cbac9390a0e06bc0f6" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33285 } }
I20260925 03:01:25.236768 4571 leader_election.cc:302] T 00000000000000000000000000000000 P 087d87d20c7945cbac9390a0e06bc0f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 087d87d20c7945cbac9390a0e06bc0f6; no voters:
I20260925 03:01:25.237315 4571 leader_election.cc:288] T 00000000000000000000000000000000 P 087d87d20c7945cbac9390a0e06bc0f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:01:25.237452 4576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 087d87d20c7945cbac9390a0e06bc0f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:01:25.238161 4576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 087d87d20c7945cbac9390a0e06bc0f6 [term 1 LEADER]: Becoming Leader. State: Replica: 087d87d20c7945cbac9390a0e06bc0f6, State: Running, Role: LEADER
I20260925 03:01:25.238679 4576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 087d87d20c7945cbac9390a0e06bc0f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087d87d20c7945cbac9390a0e06bc0f6" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33285 } }
I20260925 03:01:25.239418 4571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 087d87d20c7945cbac9390a0e06bc0f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:01:25.241181 4577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 087d87d20c7945cbac9390a0e06bc0f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "087d87d20c7945cbac9390a0e06bc0f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087d87d20c7945cbac9390a0e06bc0f6" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33285 } } }
I20260925 03:01:25.241506 4577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 087d87d20c7945cbac9390a0e06bc0f6 [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:25.243186 4582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:01:25.243072 4578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 087d87d20c7945cbac9390a0e06bc0f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 087d87d20c7945cbac9390a0e06bc0f6. Latest consensus state: current_term: 1 leader_uuid: "087d87d20c7945cbac9390a0e06bc0f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087d87d20c7945cbac9390a0e06bc0f6" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 33285 } } }
I20260925 03:01:25.243392 4578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 087d87d20c7945cbac9390a0e06bc0f6 [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:25.248247 4582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:01:25.256979 4582 catalog_manager.cc:1347] Generated new cluster ID: 77c2a917e23d4180b3a7ab4570979d5b
I20260925 03:01:25.257121 4582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:01:25.285211 4582 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:01:25.287079 4582 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:01:25.309737 4582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 087d87d20c7945cbac9390a0e06bc0f6: Generated new TSK 0
I20260925 03:01:25.310613 4582 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:01:25.326746 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33285
--builtin_ntp_servers=127.18.196.212:33063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 03:01:25.656594 4595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:25.656847 4595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:25.656988 4595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:25.664198 4595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:25.664286 4595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:25.664490 4595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 03:01:25.673167 4595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:33063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:25.674260 4595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:25.676009 4595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:25.689441 4600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:25.689518 4601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:25.689551 4603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:25.690344 4595 server_base.cc:1034] running on GCE node
I20260925 03:01:25.690826 4595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:25.691677 4595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:25.692988 4595 hybrid_clock.cc:648] HybridClock initialized: now 1790305285692937 us; error 36 us; skew 500 ppm
I20260925 03:01:25.693315 4595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:01:25.695546 4595 webserver.cc:460] Webserver started at http://127.18.196.193:34903/ using document root <none> and password file <none>
I20260925 03:01:25.696175 4595 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:25.696290 4595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:25.696601 4595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:25.699395 4595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "28a7952979c34acc9c0c7c8e0d8f41b5"
format_stamp: "Formatted at 2026-09-25 03:01:25 on dist-test-slave-8zvk"
server_key: "c4ecf70ee65ad5ded49200df3f798806"
server_key_iv: "e025c1955c9f36f9565134bd075cf271"
server_key_version: "encryptionkey@0"
I20260925 03:01:25.700197 4595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "28a7952979c34acc9c0c7c8e0d8f41b5"
format_stamp: "Formatted at 2026-09-25 03:01:25 on dist-test-slave-8zvk"
server_key: "c4ecf70ee65ad5ded49200df3f798806"
server_key_iv: "e025c1955c9f36f9565134bd075cf271"
server_key_version: "encryptionkey@0"
I20260925 03:01:25.704896 4595 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 03:01:25.708096 4609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:25.709282 4595 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 03:01:25.709460 4595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "28a7952979c34acc9c0c7c8e0d8f41b5"
format_stamp: "Formatted at 2026-09-25 03:01:25 on dist-test-slave-8zvk"
server_key: "c4ecf70ee65ad5ded49200df3f798806"
server_key_iv: "e025c1955c9f36f9565134bd075cf271"
server_key_version: "encryptionkey@0"
I20260925 03:01:25.709663 4595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:25.736598 4595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:25.737360 4595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:25.737677 4595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:25.738580 4595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:01:25.740172 4595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:01:25.740320 4595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:25.740461 4595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:01:25.740548 4595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:25.773013 4595 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:33685
I20260925 03:01:25.773104 4722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:33685 every 8 connection(s)
I20260925 03:01:25.774255 4595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 03:01:25.782099 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 4595
I20260925 03:01:25.782285 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 03:01:25.782902 19219 external_mini_cluster.cc:1428] Setting key eec6dd24cc70fff4feb82af51553a22c
I20260925 03:01:25.785990 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33285
--builtin_ntp_servers=127.18.196.212:33063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 03:01:25.787185 4725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:25.776770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:43719 (local address 127.18.196.254:33285)
0925 03:01:25.777203 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:25.777207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:25.778353 (+ 1146us) server_negotiation.cc:407] Connection header received
0925 03:01:25.780039 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:25.780043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:25.780141 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:25.780234 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:25.781187 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:25.781899 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:25.784082 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:25.784692 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:25.784825 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:25.786290 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:25.786299 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:25.786306 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:25.786344 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:25.786394 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:25.786412 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:25.786479 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:25.786812 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 03:01:25.786935 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":234,"threads_started":1}
I20260925 03:01:25.788480 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:25.777364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:33285 (local address 127.18.196.193:43719)
0925 03:01:25.778190 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:25.778233 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:25.779800 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:25.780413 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:25.780421 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:25.780492 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:01:25.781043 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:25.781063 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:25.782045 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:25.782051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:25.783948 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:25.783960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:25.784924 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:25.784932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:25.785211 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:25.785901 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 03:01:25.785906 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:01:25.785959 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 03:01:25.786054 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:01:25.786569 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:01:25.786583 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:01:25.786594 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 03:01:25.786765 (+ 171us) client_negotiation.cc:240] Negotiation successful
0925 03:01:25.787190 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":258,"threads_started":1}
I20260925 03:01:25.789959 4723 heartbeater.cc:347] Connected to a master server at 127.18.196.254:33285
I20260925 03:01:25.790236 4723 heartbeater.cc:464] Registering TS with master...
I20260925 03:01:25.791085 4723 heartbeater.cc:511] Master 127.18.196.254:33285 requested a full tablet report, sending...
I20260925 03:01:25.793040 4536 ts_manager.cc:194] Registered new tserver with Master: 28a7952979c34acc9c0c7c8e0d8f41b5 (127.18.196.193:33685)
I20260925 03:01:25.794423 4536 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.196.193:43719
W20260925 03:01:26.099558 4727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:26.099798 4727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:26.099979 4727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:26.106930 4727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:26.107018 4727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:26.107197 4727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 03:01:26.115970 4727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:33063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:26.117092 4727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:26.118727 4727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:26.131275 4735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:26.131228 4733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:26.131313 4732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:26.132040 4727 server_base.cc:1034] running on GCE node
I20260925 03:01:26.132645 4727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:26.133530 4727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:26.134744 4727 hybrid_clock.cc:648] HybridClock initialized: now 1790305286134713 us; error 46 us; skew 500 ppm
I20260925 03:01:26.135071 4727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:01:26.137053 4727 webserver.cc:460] Webserver started at http://127.18.196.194:39103/ using document root <none> and password file <none>
I20260925 03:01:26.137691 4727 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:26.137830 4727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:26.138168 4727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:26.141006 4727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "0f49902fad9844ba98f0ba112e8b724b"
format_stamp: "Formatted at 2026-09-25 03:01:26 on dist-test-slave-8zvk"
server_key: "bd959bcfd06d02c61b633871bf33e270"
server_key_iv: "df590325f78e77c92ef5f8a7d1cc31ae"
server_key_version: "encryptionkey@0"
I20260925 03:01:26.141764 4727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "0f49902fad9844ba98f0ba112e8b724b"
format_stamp: "Formatted at 2026-09-25 03:01:26 on dist-test-slave-8zvk"
server_key: "bd959bcfd06d02c61b633871bf33e270"
server_key_iv: "df590325f78e77c92ef5f8a7d1cc31ae"
server_key_version: "encryptionkey@0"
I20260925 03:01:26.146502 4727 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 03:01:26.149613 4741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:26.150668 4727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 03:01:26.150821 4727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "0f49902fad9844ba98f0ba112e8b724b"
format_stamp: "Formatted at 2026-09-25 03:01:26 on dist-test-slave-8zvk"
server_key: "bd959bcfd06d02c61b633871bf33e270"
server_key_iv: "df590325f78e77c92ef5f8a7d1cc31ae"
server_key_version: "encryptionkey@0"
I20260925 03:01:26.151072 4727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:26.179592 4727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:26.180302 4727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:26.180574 4727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:26.181428 4727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:01:26.182755 4727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:01:26.182909 4727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:26.183053 4727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:01:26.183139 4727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:26.217729 4727 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:33753
I20260925 03:01:26.217808 4854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:33753 every 8 connection(s)
I20260925 03:01:26.218978 4727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 03:01:26.220494 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 4727
I20260925 03:01:26.220669 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 03:01:26.221307 19219 external_mini_cluster.cc:1428] Setting key 97bfb1e5fa4728ec3149125b9519c85a
I20260925 03:01:26.224543 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33285
--builtin_ntp_servers=127.18.196.212:33063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 03:01:26.233472 4725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:26.222764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:54447 (local address 127.18.196.254:33285)
0925 03:01:26.223575 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:26.223579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:26.223594 (+ 15us) server_negotiation.cc:407] Connection header received
0925 03:01:26.226037 (+ 2443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:26.226041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:26.226112 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:26.226191 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:26.227515 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:26.228245 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:26.230729 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:26.231179 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:26.231276 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:26.232786 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:26.232790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:26.232793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:26.232810 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:26.232840 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:26.232852 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:26.232916 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:26.233155 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 03:01:26.233240 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694}
I20260925 03:01:26.234632 4857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:26.222213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:33285 (local address 127.18.196.194:54447)
0925 03:01:26.223295 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:26.223335 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:26.225740 (+ 2405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:26.226444 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:26.226456 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:26.226561 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:01:26.227344 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:26.227370 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:26.228402 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:26.228410 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:26.230501 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:26.230519 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:26.231394 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:26.231403 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:26.231722 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:26.232441 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 03:01:26.232446 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:01:26.232503 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 03:01:26.232595 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:01:26.232943 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:01:26.232955 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:01:26.232966 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 03:01:26.233138 (+ 172us) client_negotiation.cc:240] Negotiation successful
0925 03:01:26.233553 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":339,"threads_started":1}
I20260925 03:01:26.236083 4855 heartbeater.cc:347] Connected to a master server at 127.18.196.254:33285
I20260925 03:01:26.236369 4855 heartbeater.cc:464] Registering TS with master...
I20260925 03:01:26.237046 4855 heartbeater.cc:511] Master 127.18.196.254:33285 requested a full tablet report, sending...
I20260925 03:01:26.238399 4536 ts_manager.cc:194] Registered new tserver with Master: 0f49902fad9844ba98f0ba112e8b724b (127.18.196.194:33753)
I20260925 03:01:26.239642 4536 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.196.194:54447
W20260925 03:01:26.545897 4858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:26.546149 4858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:26.546229 4858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:26.553169 4858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:26.553299 4858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:26.553413 4858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 03:01:26.562397 4858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:33063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:33285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:26.563608 4858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:26.565268 4858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:26.577899 4863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:26.577831 4866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:26.578016 4864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:26.578797 4858 server_base.cc:1034] running on GCE node
I20260925 03:01:26.579484 4858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:26.580364 4858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:26.581557 4858 hybrid_clock.cc:648] HybridClock initialized: now 1790305286581498 us; error 49 us; skew 500 ppm
I20260925 03:01:26.581863 4858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:01:26.584022 4858 webserver.cc:460] Webserver started at http://127.18.196.195:40199/ using document root <none> and password file <none>
I20260925 03:01:26.584669 4858 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:26.584779 4858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:26.585088 4858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:26.587819 4858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "a02b9973d49d43ca8f8274bc044860fb"
format_stamp: "Formatted at 2026-09-25 03:01:26 on dist-test-slave-8zvk"
server_key: "4fb27af5568b5851b7d6d9ac75d5f95c"
server_key_iv: "1ce0ae4dd082821ace72490606921d09"
server_key_version: "encryptionkey@0"
I20260925 03:01:26.588585 4858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "a02b9973d49d43ca8f8274bc044860fb"
format_stamp: "Formatted at 2026-09-25 03:01:26 on dist-test-slave-8zvk"
server_key: "4fb27af5568b5851b7d6d9ac75d5f95c"
server_key_iv: "1ce0ae4dd082821ace72490606921d09"
server_key_version: "encryptionkey@0"
I20260925 03:01:26.593379 4858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 03:01:26.596665 4872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:26.597921 4858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 03:01:26.598110 4858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "a02b9973d49d43ca8f8274bc044860fb"
format_stamp: "Formatted at 2026-09-25 03:01:26 on dist-test-slave-8zvk"
server_key: "4fb27af5568b5851b7d6d9ac75d5f95c"
server_key_iv: "1ce0ae4dd082821ace72490606921d09"
server_key_version: "encryptionkey@0"
I20260925 03:01:26.598331 4858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:26.621519 4858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:26.622400 4858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:26.622702 4858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:26.623626 4858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:01:26.625041 4858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:01:26.625164 4858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:26.625300 4858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:01:26.625389 4858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:26.658100 4858 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:35025
I20260925 03:01:26.658183 4984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:35025 every 8 connection(s)
I20260925 03:01:26.659428 4858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 03:01:26.662186 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 4858
I20260925 03:01:26.662371 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 03:01:26.662775 19219 external_mini_cluster.cc:1428] Setting key 659850df7ca1727b9dfcf3865fffd376
I20260925 03:01:26.671600 4725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:26.662056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:40447 (local address 127.18.196.254:33285)
0925 03:01:26.662281 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:26.662287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:26.663791 (+ 1504us) server_negotiation.cc:407] Connection header received
0925 03:01:26.665896 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:26.665901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:26.665966 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:26.666046 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:26.667070 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:26.667811 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:26.669261 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:26.669663 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:26.669753 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:26.670961 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:26.670965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:26.670969 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:26.670986 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:26.671017 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:26.671029 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:26.671098 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:26.671295 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 03:01:26.671366 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 03:01:26.672463 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:26.662633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:33285 (local address 127.18.196.195:40447)
0925 03:01:26.663625 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:26.663670 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:26.665668 (+ 1998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:26.666233 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:26.666240 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:26.666323 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:01:26.666897 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:26.666922 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:26.667958 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:26.667962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:26.669139 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:26.669151 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:26.669845 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:26.669849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:26.670032 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:26.670608 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 03:01:26.670610 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:01:26.670645 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 03:01:26.670714 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:01:26.671115 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:01:26.671123 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:01:26.671128 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 03:01:26.671266 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 03:01:26.671531 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":318,"threads_started":1}
I20260925 03:01:26.673659 4985 heartbeater.cc:347] Connected to a master server at 127.18.196.254:33285
I20260925 03:01:26.673900 4985 heartbeater.cc:464] Registering TS with master...
I20260925 03:01:26.674522 4985 heartbeater.cc:511] Master 127.18.196.254:33285 requested a full tablet report, sending...
I20260925 03:01:26.675829 4536 ts_manager.cc:194] Registered new tserver with Master: a02b9973d49d43ca8f8274bc044860fb (127.18.196.195:35025)
I20260925 03:01:26.676641 4536 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.196.195:40447
I20260925 03:01:26.677680 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 03:01:26.688100 4725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:26.681581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46692 (local address 127.18.196.254:33285)
0925 03:01:26.681727 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:26.681740 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:26.682249 (+ 509us) server_negotiation.cc:407] Connection header received
0925 03:01:26.682378 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:26.682381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:26.682448 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:26.682524 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:26.683004 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:26.683656 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:26.685318 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:26.685944 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:26.686090 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:26.687389 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:26.687394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:26.687397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:26.687417 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:26.687446 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:26.687457 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:26.687526 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:26.687705 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 03:01:26.687794 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 03:01:26.691479 4536 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 03:01:26.698158 4536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 03:01:26.727392 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:26.709399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33685 (local address 127.0.0.1:50702)
0925 03:01:26.710948 (+ 1549us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:26.711002 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:26.711208 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:26.719159 (+ 7951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:26.719166 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:26.719195 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:26.719810 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:26.719832 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:26.721872 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:26.721877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:26.724435 (+ 2558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:26.724460 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:26.726322 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:26.726329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:26.726545 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:26.726607 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 03:01:26.726711 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 03:01:26.726925 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1238,"mutex_wait_us":24,"thread_start_us":8393,"threads_started":1}
I20260925 03:01:26.727452 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:26.710098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:33753 (local address 127.0.0.1:53320)
0925 03:01:26.718068 (+ 7970us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:26.718085 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:26.718978 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:26.719911 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:26.719914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:26.719933 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:26.720394 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:26.720411 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:26.722338 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:26.722348 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:26.724571 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:26.724586 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:26.726572 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:26.726577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:26.726692 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:26.726714 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 03:01:26.726828 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 03:01:26.726973 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7846,"mutex_wait_us":52,"thread_start_us":383,"threads_started":1}
I20260925 03:01:26.727941 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:26.710504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53320 (local address 127.18.196.194:33753)
0925 03:01:26.717659 (+ 7155us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:26.717668 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:26.718175 (+ 507us) server_negotiation.cc:407] Connection header received
0925 03:01:26.719178 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:26.719186 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:26.719408 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:26.719582 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:26.720591 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:26.722200 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:26.724756 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:26.726063 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:26.726272 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:26.726462 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:26.726903 (+ 441us) server_negotiation.cc:299] Negotiation successful
0925 03:01:26.727063 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6977,"thread_start_us":7654,"threads_started":1}
I20260925 03:01:26.728147 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:26.718581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56120 (local address 127.18.196.195:35025)
0925 03:01:26.719281 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:26.719296 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:26.720468 (+ 1172us) server_negotiation.cc:407] Connection header received
0925 03:01:26.720724 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:26.720730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:26.720949 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:26.721130 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:26.722110 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:26.723220 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:26.725176 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:26.726563 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:26.726721 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:26.726962 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:26.727594 (+ 632us) server_negotiation.cc:299] Negotiation successful
0925 03:01:26.727775 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":408,"threads_started":1}
I20260925 03:01:26.728677 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:26.709679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50702 (local address 127.18.196.193:33685)
0925 03:01:26.718561 (+ 8882us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:26.718567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:26.718588 (+ 21us) server_negotiation.cc:407] Connection header received
0925 03:01:26.718664 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:26.718669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:26.718913 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:26.719667 (+ 754us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:26.720055 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:26.721643 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:26.724668 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:26.726106 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:26.727889 (+ 1783us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:26.728095 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:26.728169 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 03:01:26.728331 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8751,"thread_start_us":8628,"threads_started":1}
I20260925 03:01:26.729276 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:26.719298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:35025 (local address 127.0.0.1:56120)
0925 03:01:26.720351 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:26.720368 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:26.720510 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:26.721425 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:26.721429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:26.721453 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:26.721885 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:26.721897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:26.723355 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:26.723363 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:26.724954 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:26.724969 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:26.727139 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:26.727144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:26.727351 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:26.727412 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 03:01:26.728873 (+ 1461us) client_negotiation.cc:240] Negotiation successful
0925 03:01:26.728997 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"thread_start_us":791,"threads_started":1}
I20260925 03:01:26.732177 4790 tablet_service.cc:1467] Processing CreateTablet for tablet acd5ce999a6d4a35aac9ecdbfc1b9137 (DEFAULT_TABLE table=test-table [id=ce0e58edbcbd49ebbb3ca6e403858885]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 03:01:26.732177 4657 tablet_service.cc:1467] Processing CreateTablet for tablet acd5ce999a6d4a35aac9ecdbfc1b9137 (DEFAULT_TABLE table=test-table [id=ce0e58edbcbd49ebbb3ca6e403858885]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 03:01:26.732496 4920 tablet_service.cc:1467] Processing CreateTablet for tablet acd5ce999a6d4a35aac9ecdbfc1b9137 (DEFAULT_TABLE table=test-table [id=ce0e58edbcbd49ebbb3ca6e403858885]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 03:01:26.733629 4790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acd5ce999a6d4a35aac9ecdbfc1b9137. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:26.733629 4657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acd5ce999a6d4a35aac9ecdbfc1b9137. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:26.733716 4920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acd5ce999a6d4a35aac9ecdbfc1b9137. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:26.745891 5004 tablet_bootstrap.cc:492] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb: Bootstrap starting.
I20260925 03:01:26.748718 5006 tablet_bootstrap.cc:492] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 28a7952979c34acc9c0c7c8e0d8f41b5: Bootstrap starting.
I20260925 03:01:26.750120 5004 tablet_bootstrap.cc:654] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:26.750905 5005 tablet_bootstrap.cc:492] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 0f49902fad9844ba98f0ba112e8b724b: Bootstrap starting.
I20260925 03:01:26.751407 5004 log.cc:824] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:26.752208 5006 tablet_bootstrap.cc:654] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 28a7952979c34acc9c0c7c8e0d8f41b5: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:26.753340 5006 log.cc:824] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 28a7952979c34acc9c0c7c8e0d8f41b5: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:26.754118 5005 tablet_bootstrap.cc:654] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 0f49902fad9844ba98f0ba112e8b724b: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:26.755331 5005 log.cc:824] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 0f49902fad9844ba98f0ba112e8b724b: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:26.755502 5004 tablet_bootstrap.cc:492] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb: No bootstrap required, opened a new log
I20260925 03:01:26.755772 5004 ts_tablet_manager.cc:1397] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260925 03:01:26.756522 5006 tablet_bootstrap.cc:492] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 28a7952979c34acc9c0c7c8e0d8f41b5: No bootstrap required, opened a new log
I20260925 03:01:26.756765 5006 ts_tablet_manager.cc:1397] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 28a7952979c34acc9c0c7c8e0d8f41b5: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 03:01:26.757754 5005 tablet_bootstrap.cc:492] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 0f49902fad9844ba98f0ba112e8b724b: No bootstrap required, opened a new log
I20260925 03:01:26.757992 5005 ts_tablet_manager.cc:1397] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 0f49902fad9844ba98f0ba112e8b724b: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 03:01:26.765321 5005 raft_consensus.cc:348] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 0f49902fad9844ba98f0ba112e8b724b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a7952979c34acc9c0c7c8e0d8f41b5" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33685 } } peers { permanent_uuid: "a02b9973d49d43ca8f8274bc044860fb" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 35025 } } peers { permanent_uuid: "0f49902fad9844ba98f0ba112e8b724b" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 33753 } }
I20260925 03:01:26.765683 5005 raft_consensus.cc:374] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 0f49902fad9844ba98f0ba112e8b724b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:26.765774 5005 raft_consensus.cc:729] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 0f49902fad9844ba98f0ba112e8b724b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f49902fad9844ba98f0ba112e8b724b, State: Initialized, Role: FOLLOWER
I20260925 03:01:26.766348 5005 consensus_queue.cc:260] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 0f49902fad9844ba98f0ba112e8b724b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a7952979c34acc9c0c7c8e0d8f41b5" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33685 } } peers { permanent_uuid: "a02b9973d49d43ca8f8274bc044860fb" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 35025 } } peers { permanent_uuid: "0f49902fad9844ba98f0ba112e8b724b" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 33753 } }
I20260925 03:01:26.766986 5006 raft_consensus.cc:348] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 28a7952979c34acc9c0c7c8e0d8f41b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a7952979c34acc9c0c7c8e0d8f41b5" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33685 } } peers { permanent_uuid: "a02b9973d49d43ca8f8274bc044860fb" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 35025 } } peers { permanent_uuid: "0f49902fad9844ba98f0ba112e8b724b" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 33753 } }
I20260925 03:01:26.766986 5004 raft_consensus.cc:348] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a7952979c34acc9c0c7c8e0d8f41b5" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33685 } } peers { permanent_uuid: "a02b9973d49d43ca8f8274bc044860fb" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 35025 } } peers { permanent_uuid: "0f49902fad9844ba98f0ba112e8b724b" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 33753 } }
I20260925 03:01:26.767405 5006 raft_consensus.cc:374] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 28a7952979c34acc9c0c7c8e0d8f41b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:26.767395 5004 raft_consensus.cc:374] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:26.767448 4855 heartbeater.cc:503] Master 127.18.196.254:33285 was elected leader, sending a full tablet report...
I20260925 03:01:26.767491 5004 raft_consensus.cc:729] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a02b9973d49d43ca8f8274bc044860fb, State: Initialized, Role: FOLLOWER
I20260925 03:01:26.767491 5006 raft_consensus.cc:729] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 28a7952979c34acc9c0c7c8e0d8f41b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a7952979c34acc9c0c7c8e0d8f41b5, State: Initialized, Role: FOLLOWER
I20260925 03:01:26.768123 5005 ts_tablet_manager.cc:1428] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 0f49902fad9844ba98f0ba112e8b724b: Time spent starting tablet: real 0.010s user 0.002s sys 0.008s
I20260925 03:01:26.768155 5004 consensus_queue.cc:260] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a7952979c34acc9c0c7c8e0d8f41b5" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33685 } } peers { permanent_uuid: "a02b9973d49d43ca8f8274bc044860fb" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 35025 } } peers { permanent_uuid: "0f49902fad9844ba98f0ba112e8b724b" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 33753 } }
I20260925 03:01:26.768155 5006 consensus_queue.cc:260] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 28a7952979c34acc9c0c7c8e0d8f41b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a7952979c34acc9c0c7c8e0d8f41b5" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33685 } } peers { permanent_uuid: "a02b9973d49d43ca8f8274bc044860fb" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 35025 } } peers { permanent_uuid: "0f49902fad9844ba98f0ba112e8b724b" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 33753 } }
I20260925 03:01:26.769636 4723 heartbeater.cc:503] Master 127.18.196.254:33285 was elected leader, sending a full tablet report...
I20260925 03:01:26.769670 4985 heartbeater.cc:503] Master 127.18.196.254:33285 was elected leader, sending a full tablet report...
I20260925 03:01:26.770195 5006 ts_tablet_manager.cc:1428] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 28a7952979c34acc9c0c7c8e0d8f41b5: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260925 03:01:26.771207 5004 ts_tablet_manager.cc:1428] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260925 03:01:26.776484 4724 tablet.cc:2369] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 28a7952979c34acc9c0c7c8e0d8f41b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 03:01:26.911314 4986 tablet.cc:2369] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 03:01:26.971067 4856 tablet.cc:2369] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 0f49902fad9844ba98f0ba112e8b724b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 03:01:27.070169 5011 raft_consensus.cc:482] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:01:27.070536 5011 raft_consensus.cc:504] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a7952979c34acc9c0c7c8e0d8f41b5" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33685 } } peers { permanent_uuid: "a02b9973d49d43ca8f8274bc044860fb" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 35025 } } peers { permanent_uuid: "0f49902fad9844ba98f0ba112e8b724b" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 33753 } }
I20260925 03:01:27.072320 5011 leader_election.cc:288] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a7952979c34acc9c0c7c8e0d8f41b5 (127.18.196.193:33685), 0f49902fad9844ba98f0ba112e8b724b (127.18.196.194:33753)
I20260925 03:01:27.079255 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:27.073025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:57239 (local address 127.18.196.194:33753)
0925 03:01:27.073269 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:27.073286 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:27.073307 (+ 21us) server_negotiation.cc:407] Connection header received
0925 03:01:27.073391 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:27.073399 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:27.073480 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:27.073582 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:27.074178 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:27.075202 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:27.077383 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:27.078316 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:27.078439 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:27.078486 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:27.078840 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 03:01:27.078999 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260925 03:01:27.079300 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:27.072590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:33753 (local address 127.18.196.195:57239)
0925 03:01:27.072794 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:27.072815 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:27.072947 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:27.073639 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:27.073643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:27.073669 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:27.073974 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:27.073986 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:27.075332 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:27.075337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:27.077222 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:27.077237 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:27.078475 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:27.078480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:27.078644 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:27.078680 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 03:01:27.078784 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 03:01:27.078970 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"spinlock_wait_cycles":21760}
I20260925 03:01:27.079650 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:27.072839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33685 (local address 127.18.196.195:56825)
0925 03:01:27.073626 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:27.073645 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:27.073790 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:27.074160 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:27.074164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:27.074188 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:27.074580 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:27.074591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:27.075793 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:27.075798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:27.077654 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:27.077668 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:27.078905 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:27.078911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:27.079097 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:27.079134 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 03:01:27.079228 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 03:01:27.079380 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":393,"threads_started":1}
I20260925 03:01:27.079777 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:27.072897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:56825 (local address 127.18.196.193:33685)
0925 03:01:27.073134 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:27.073143 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:27.073746 (+ 603us) server_negotiation.cc:407] Connection header received
0925 03:01:27.073937 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:27.073942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:27.074035 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:27.074131 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:27.074747 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:27.075565 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:27.077844 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:27.078723 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:27.078878 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:27.078955 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:27.079278 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 03:01:27.079428 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 03:01:27.080519 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd5ce999a6d4a35aac9ecdbfc1b9137" candidate_uuid: "a02b9973d49d43ca8f8274bc044860fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f49902fad9844ba98f0ba112e8b724b" is_pre_election: true
I20260925 03:01:27.080677 4678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd5ce999a6d4a35aac9ecdbfc1b9137" candidate_uuid: "a02b9973d49d43ca8f8274bc044860fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a7952979c34acc9c0c7c8e0d8f41b5" is_pre_election: true
I20260925 03:01:27.081061 4810 raft_consensus.cc:2454] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 0f49902fad9844ba98f0ba112e8b724b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a02b9973d49d43ca8f8274bc044860fb in term 0.
I20260925 03:01:27.081238 4678 raft_consensus.cc:2454] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 28a7952979c34acc9c0c7c8e0d8f41b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a02b9973d49d43ca8f8274bc044860fb in term 0.
I20260925 03:01:27.081699 4874 leader_election.cc:302] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f49902fad9844ba98f0ba112e8b724b, a02b9973d49d43ca8f8274bc044860fb; no voters:
I20260925 03:01:27.082093 5011 raft_consensus.cc:2781] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:01:27.082242 5011 raft_consensus.cc:482] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:01:27.082345 5011 raft_consensus.cc:3037] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:27.084622 5011 raft_consensus.cc:504] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a7952979c34acc9c0c7c8e0d8f41b5" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33685 } } peers { permanent_uuid: "a02b9973d49d43ca8f8274bc044860fb" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 35025 } } peers { permanent_uuid: "0f49902fad9844ba98f0ba112e8b724b" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 33753 } }
I20260925 03:01:27.085179 5011 leader_election.cc:288] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb [CANDIDATE]: Term 1 election: Requested vote from peers 28a7952979c34acc9c0c7c8e0d8f41b5 (127.18.196.193:33685), 0f49902fad9844ba98f0ba112e8b724b (127.18.196.194:33753)
I20260925 03:01:27.085614 4678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd5ce999a6d4a35aac9ecdbfc1b9137" candidate_uuid: "a02b9973d49d43ca8f8274bc044860fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a7952979c34acc9c0c7c8e0d8f41b5"
I20260925 03:01:27.085673 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd5ce999a6d4a35aac9ecdbfc1b9137" candidate_uuid: "a02b9973d49d43ca8f8274bc044860fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f49902fad9844ba98f0ba112e8b724b"
I20260925 03:01:27.085809 4678 raft_consensus.cc:3037] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 28a7952979c34acc9c0c7c8e0d8f41b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:27.085891 4810 raft_consensus.cc:3037] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 0f49902fad9844ba98f0ba112e8b724b [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:27.088078 4678 raft_consensus.cc:2454] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 28a7952979c34acc9c0c7c8e0d8f41b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a02b9973d49d43ca8f8274bc044860fb in term 1.
I20260925 03:01:27.088145 4810 raft_consensus.cc:2454] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 0f49902fad9844ba98f0ba112e8b724b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a02b9973d49d43ca8f8274bc044860fb in term 1.
I20260925 03:01:27.088528 4875 leader_election.cc:302] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a7952979c34acc9c0c7c8e0d8f41b5, a02b9973d49d43ca8f8274bc044860fb; no voters:
I20260925 03:01:27.088852 5011 raft_consensus.cc:2781] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:01:27.089257 5011 raft_consensus.cc:686] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb [term 1 LEADER]: Becoming Leader. State: Replica: a02b9973d49d43ca8f8274bc044860fb, State: Running, Role: LEADER
I20260925 03:01:27.089692 5011 consensus_queue.cc:237] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a7952979c34acc9c0c7c8e0d8f41b5" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33685 } } peers { permanent_uuid: "a02b9973d49d43ca8f8274bc044860fb" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 35025 } } peers { permanent_uuid: "0f49902fad9844ba98f0ba112e8b724b" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 33753 } }
I20260925 03:01:27.094125 4534 catalog_manager.cc:5495] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb reported cstate change: term changed from 0 to 1, leader changed from <none> to a02b9973d49d43ca8f8274bc044860fb (127.18.196.195). New cstate: current_term: 1 leader_uuid: "a02b9973d49d43ca8f8274bc044860fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a7952979c34acc9c0c7c8e0d8f41b5" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a02b9973d49d43ca8f8274bc044860fb" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f49902fad9844ba98f0ba112e8b724b" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 33753 } health_report { overall_health: UNKNOWN } } }
I20260925 03:01:27.177052 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:27.171912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56136 (local address 127.18.196.195:35025)
0925 03:01:27.172085 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:27.172089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:27.172248 (+ 159us) server_negotiation.cc:407] Connection header received
0925 03:01:27.172416 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:27.172420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:27.172511 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:27.172637 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:01:27.173065 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:27.173810 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:27.175211 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:27.175603 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:27.175696 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:27.176118 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:01:27.176598 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:01:27.176705 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:27.176801 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 03:01:27.176863 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 03:01:27.185148 4678 raft_consensus.cc:1261] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 28a7952979c34acc9c0c7c8e0d8f41b5 [term 1 FOLLOWER]: Refusing update from remote peer a02b9973d49d43ca8f8274bc044860fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:01:27.185144 4810 raft_consensus.cc:1261] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 0f49902fad9844ba98f0ba112e8b724b [term 1 FOLLOWER]: Refusing update from remote peer a02b9973d49d43ca8f8274bc044860fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:01:27.186116 5011 consensus_queue.cc:1035] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f49902fad9844ba98f0ba112e8b724b" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 33753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:01:27.186415 5014 consensus_queue.cc:1035] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a7952979c34acc9c0c7c8e0d8f41b5" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:01:27.211863 4535 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 03:01:27.212127 4535 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 03:01:27.231521 4535 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 087d87d20c7945cbac9390a0e06bc0f6: Sending DeleteTablet for 3 replicas of tablet acd5ce999a6d4a35aac9ecdbfc1b9137
I20260925 03:01:27.233018 4790 tablet_service.cc:1514] Processing DeleteTablet for tablet acd5ce999a6d4a35aac9ecdbfc1b9137 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 03:01:27 UTC) from {username='slave'} at 127.0.0.1:53320
I20260925 03:01:27.233076 4657 tablet_service.cc:1514] Processing DeleteTablet for tablet acd5ce999a6d4a35aac9ecdbfc1b9137 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 03:01:27 UTC) from {username='slave'} at 127.0.0.1:50702
I20260925 03:01:27.233893 5027 tablet_replica.cc:330] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 28a7952979c34acc9c0c7c8e0d8f41b5: stopping tablet replica
I20260925 03:01:27.233754 4920 tablet_service.cc:1514] Processing DeleteTablet for tablet acd5ce999a6d4a35aac9ecdbfc1b9137 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 03:01:27 UTC) from {username='slave'} at 127.0.0.1:56120
I20260925 03:01:27.234370 5027 raft_consensus.cc:2229] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 28a7952979c34acc9c0c7c8e0d8f41b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:01:27.234992 5027 raft_consensus.cc:2258] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 28a7952979c34acc9c0c7c8e0d8f41b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:01:27.235173 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 4595
I20260925 03:01:27.237697 5027 ts_tablet_manager.cc:1905] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 28a7952979c34acc9c0c7c8e0d8f41b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 03:01:27.239717 5029 tablet_replica.cc:330] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb: stopping tablet replica
I20260925 03:01:27.239846 5028 tablet_replica.cc:330] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 0f49902fad9844ba98f0ba112e8b724b: stopping tablet replica
I20260925 03:01:27.240178 5028 raft_consensus.cc:2229] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 0f49902fad9844ba98f0ba112e8b724b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:01:27.240231 5029 raft_consensus.cc:2229] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb [term 1 LEADER]: Raft consensus shutting down.
I20260925 03:01:27.240666 5028 raft_consensus.cc:2258] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 0f49902fad9844ba98f0ba112e8b724b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:01:27.240971 5029 raft_consensus.cc:2258] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:01:27.244470 5028 ts_tablet_manager.cc:1905] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 0f49902fad9844ba98f0ba112e8b724b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 03:01:27.244897 5029 ts_tablet_manager.cc:1905] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 03:01:27.251711 5027 ts_tablet_manager.cc:1918] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 28a7952979c34acc9c0c7c8e0d8f41b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 03:01:27.251721 5028 ts_tablet_manager.cc:1918] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 0f49902fad9844ba98f0ba112e8b724b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 03:01:27.251886 5028 log.cc:1196] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 0f49902fad9844ba98f0ba112e8b724b: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/acd5ce999a6d4a35aac9ecdbfc1b9137
I20260925 03:01:27.251891 5027 log.cc:1196] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 28a7952979c34acc9c0c7c8e0d8f41b5: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-0/wal/wals/acd5ce999a6d4a35aac9ecdbfc1b9137
I20260925 03:01:27.252460 5027 ts_tablet_manager.cc:1939] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 28a7952979c34acc9c0c7c8e0d8f41b5: Deleting consensus metadata
I20260925 03:01:27.252624 5028 ts_tablet_manager.cc:1939] T acd5ce999a6d4a35aac9ecdbfc1b9137 P 0f49902fad9844ba98f0ba112e8b724b: Deleting consensus metadata
I20260925 03:01:27.254372 4522 catalog_manager.cc:4841] TS 28a7952979c34acc9c0c7c8e0d8f41b5 (127.18.196.193:33685): tablet acd5ce999a6d4a35aac9ecdbfc1b9137 (table test-table [id=ce0e58edbcbd49ebbb3ca6e403858885]) successfully deleted
I20260925 03:01:27.254381 4521 catalog_manager.cc:4841] TS 0f49902fad9844ba98f0ba112e8b724b (127.18.196.194:33753): tablet acd5ce999a6d4a35aac9ecdbfc1b9137 (table test-table [id=ce0e58edbcbd49ebbb3ca6e403858885]) successfully deleted
I20260925 03:01:27.259588 5029 ts_tablet_manager.cc:1918] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 03:01:27.259831 5029 log.cc:1196] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/acd5ce999a6d4a35aac9ecdbfc1b9137
I20260925 03:01:27.260403 5029 ts_tablet_manager.cc:1939] T acd5ce999a6d4a35aac9ecdbfc1b9137 P a02b9973d49d43ca8f8274bc044860fb: Deleting consensus metadata
I20260925 03:01:27.262229 4522 catalog_manager.cc:4841] TS a02b9973d49d43ca8f8274bc044860fb (127.18.196.195:35025): tablet acd5ce999a6d4a35aac9ecdbfc1b9137 (table test-table [id=ce0e58edbcbd49ebbb3ca6e403858885]) successfully deleted
I20260925 03:01:27.262526 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:27.237083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50710 (local address 127.18.196.193:33685)
0925 03:01:27.237262 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:27.237270 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:27.252806 (+ 15536us) server_negotiation.cc:407] Connection header received
0925 03:01:27.253291 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:27.253297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:27.253412 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:27.253511 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:27.255042 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:27.256225 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:27.258209 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:27.258910 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:27.259012 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:27.260598 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:27.260612 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:27.260629 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:27.261468 (+ 839us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:27.261529 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:27.261557 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:27.261651 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:27.262045 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 03:01:27.262165 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 03:01:27.263603 4718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:27.351562 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 4595
I20260925 03:01:27.375542 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 4727
I20260925 03:01:27.383603 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:27.376375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53326 (local address 127.18.196.194:33753)
0925 03:01:27.376550 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:27.376555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:27.376724 (+ 169us) server_negotiation.cc:407] Connection header received
0925 03:01:27.376878 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:27.376884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:27.376997 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:27.377123 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:27.377491 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:27.378565 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:27.379882 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:27.380523 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:27.380655 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:27.381769 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:27.381783 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:27.381808 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:27.382585 (+ 777us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:27.382656 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:27.382681 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:27.382813 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:27.383141 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 03:01:27.383245 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 03:01:27.384275 4850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:27.470038 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 4727
I20260925 03:01:27.492843 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 4858
I20260925 03:01:27.500916 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:27.493604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56138 (local address 127.18.196.195:35025)
0925 03:01:27.493772 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:27.493777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:27.493855 (+ 78us) server_negotiation.cc:407] Connection header received
0925 03:01:27.494050 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:27.494054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:27.494130 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:27.494209 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:27.494752 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:27.495615 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:27.497631 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:27.498060 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:27.498159 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:27.499487 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:27.499496 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:27.499510 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:27.500098 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:27.500151 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:27.500171 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:27.500258 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:27.500442 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 03:01:27.500523 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 03:01:27.501585 4980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:27.621337 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 4858
I20260925 03:01:27.640870 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 4505
I20260925 03:01:27.641974 4566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:27.752673 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 4505
2026-09-25T03:01:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3051 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 03:01:27 dist-test-slave-8zvk krb5kdc[5037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 03:01:27 dist-test-slave-8zvk krb5kdc[5037](info): set up 2 sockets
Sep 25 03:01:27 dist-test-slave-8zvk krb5kdc[5037](info): commencing operation
krb5kdc: starting...
W20260925 03:01:28.916260 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 03:01:29 dist-test-slave-8zvk krb5kdc[5037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T03:01:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:01:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 03:01:29.306725 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:36937
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:34303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:36937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:01:29.652163 5050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:29.652421 5050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:29.652509 5050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:29.659358 5050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:01:29.659444 5050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:29.659499 5050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:29.659555 5050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:01:29.659598 5050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:01:29.668820 5050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:34303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:36937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:36937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:29.669989 5050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:29.671195 5050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:29.682633 5055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:29.682726 5056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:29.682632 5058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:29.683691 5050 server_base.cc:1034] running on GCE node
I20260925 03:01:29.684399 5050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:29.685691 5050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:29.686936 5050 hybrid_clock.cc:648] HybridClock initialized: now 1790305289686896 us; error 51 us; skew 500 ppm
I20260925 03:01:29.687253 5050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:29 dist-test-slave-8zvk krb5kdc[5037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305289, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:29.690110 5050 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 03:01:29.691715 5050 webserver.cc:460] Webserver started at http://127.18.196.254:46389/ using document root <none> and password file <none>
I20260925 03:01:29.692407 5050 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:29.692564 5050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:29.692875 5050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:29.695937 5050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "f2199ae0a5d3403a8c3760c75d6068aa"
format_stamp: "Formatted at 2026-09-25 03:01:29 on dist-test-slave-8zvk"
server_key: "7afb558f43db4cc5c620560f942aaff9"
server_key_iv: "f25d45031d8fa9c8c774f9e3021bc621"
server_key_version: "encryptionkey@0"
I20260925 03:01:29.696897 5050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "f2199ae0a5d3403a8c3760c75d6068aa"
format_stamp: "Formatted at 2026-09-25 03:01:29 on dist-test-slave-8zvk"
server_key: "7afb558f43db4cc5c620560f942aaff9"
server_key_iv: "f25d45031d8fa9c8c774f9e3021bc621"
server_key_version: "encryptionkey@0"
I20260925 03:01:29.702653 5050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 03:01:29.706013 5065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:29.707289 5050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 03:01:29.707496 5050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "f2199ae0a5d3403a8c3760c75d6068aa"
format_stamp: "Formatted at 2026-09-25 03:01:29 on dist-test-slave-8zvk"
server_key: "7afb558f43db4cc5c620560f942aaff9"
server_key_iv: "f25d45031d8fa9c8c774f9e3021bc621"
server_key_version: "encryptionkey@0"
I20260925 03:01:29.707764 5050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:29.728612 5050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:29.732401 5050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:29.732679 5050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:29.750891 5050 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:36937
I20260925 03:01:29.750972 5116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:36937 every 8 connection(s)
I20260925 03:01:29.752199 5050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:01:29.755828 5117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:29.759701 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 5050
I20260925 03:01:29.759892 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:01:29.760459 19219 external_mini_cluster.cc:1428] Setting key 50d17fa569f166efec0a7c25be0085d3
I20260925 03:01:29.768705 5117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2199ae0a5d3403a8c3760c75d6068aa: Bootstrap starting.
I20260925 03:01:29.772444 5117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2199ae0a5d3403a8c3760c75d6068aa: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:29.773746 5117 log.cc:824] T 00000000000000000000000000000000 P f2199ae0a5d3403a8c3760c75d6068aa: Log is configured to *not* fsync() on all Append() calls
Sep 25 03:01:29 dist-test-slave-8zvk krb5kdc[5037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:01:29.776646 5117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2199ae0a5d3403a8c3760c75d6068aa: No bootstrap required, opened a new log
I20260925 03:01:29.783308 5120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:29.762322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36536 (local address 127.18.196.254:36937)
0925 03:01:29.763117 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:29.763128 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:29.763168 (+ 40us) server_negotiation.cc:407] Connection header received
0925 03:01:29.764731 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:29.764752 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:29.765154 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:29.765466 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:29.767194 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:29.768412 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:29.770223 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:29.770974 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:29.771100 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:29.776139 (+ 5039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:29.776146 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:29.776159 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:29.776195 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:29.779179 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:29.779885 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:29.779891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:29.779897 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:29.780066 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:29.780458 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:29.780463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:29.780466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:29.780970 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:29.781282 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:29.781760 (+ 478us) server_negotiation.cc:299] Negotiation successful
0925 03:01:29.782201 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":366,"threads_started":1}
I20260925 03:01:29.784483 5117 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2199ae0a5d3403a8c3760c75d6068aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2199ae0a5d3403a8c3760c75d6068aa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36937 } }
I20260925 03:01:29.784907 5117 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2199ae0a5d3403a8c3760c75d6068aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:29.785039 5117 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2199ae0a5d3403a8c3760c75d6068aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2199ae0a5d3403a8c3760c75d6068aa, State: Initialized, Role: FOLLOWER
I20260925 03:01:29.785624 5117 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2199ae0a5d3403a8c3760c75d6068aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2199ae0a5d3403a8c3760c75d6068aa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36937 } }
I20260925 03:01:29.785890 5117 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2199ae0a5d3403a8c3760c75d6068aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:01:29.786072 5117 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2199ae0a5d3403a8c3760c75d6068aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:01:29.786242 5117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2199ae0a5d3403a8c3760c75d6068aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:29.788573 5117 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2199ae0a5d3403a8c3760c75d6068aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2199ae0a5d3403a8c3760c75d6068aa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36937 } }
I20260925 03:01:29.788971 5117 leader_election.cc:302] T 00000000000000000000000000000000 P f2199ae0a5d3403a8c3760c75d6068aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2199ae0a5d3403a8c3760c75d6068aa; no voters:
I20260925 03:01:29.789511 5117 leader_election.cc:288] T 00000000000000000000000000000000 P f2199ae0a5d3403a8c3760c75d6068aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:01:29.789595 5122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2199ae0a5d3403a8c3760c75d6068aa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:01:29.790242 5122 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2199ae0a5d3403a8c3760c75d6068aa [term 1 LEADER]: Becoming Leader. State: Replica: f2199ae0a5d3403a8c3760c75d6068aa, State: Running, Role: LEADER
I20260925 03:01:29.790783 5122 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2199ae0a5d3403a8c3760c75d6068aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2199ae0a5d3403a8c3760c75d6068aa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36937 } }
I20260925 03:01:29.791576 5117 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2199ae0a5d3403a8c3760c75d6068aa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:01:29.793567 5123 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2199ae0a5d3403a8c3760c75d6068aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2199ae0a5d3403a8c3760c75d6068aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2199ae0a5d3403a8c3760c75d6068aa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36937 } } }
I20260925 03:01:29.793942 5123 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2199ae0a5d3403a8c3760c75d6068aa [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:29.795890 5124 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2199ae0a5d3403a8c3760c75d6068aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2199ae0a5d3403a8c3760c75d6068aa. Latest consensus state: current_term: 1 leader_uuid: "f2199ae0a5d3403a8c3760c75d6068aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2199ae0a5d3403a8c3760c75d6068aa" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 36937 } } }
I20260925 03:01:29.796339 5129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:01:29.796343 5124 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2199ae0a5d3403a8c3760c75d6068aa [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:29.801204 5129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:01:29.809697 5129 catalog_manager.cc:1347] Generated new cluster ID: fa6ea2712ba94b0f9b9234ea122659ed
I20260925 03:01:29.809798 5129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:01:29.828792 5129 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:01:29.830607 5129 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:01:29.845057 5129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2199ae0a5d3403a8c3760c75d6068aa: Generated new TSK 0
I20260925 03:01:29.846086 5129 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
I20260925 03:01:29.911417 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:36937
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:34303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:01:30.231456 5143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:30.231707 5143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:30.231895 5143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:30.238690 5143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:30.238809 5143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:30.238996 5143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 03:01:30.248217 5143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:34303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:36937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:30.249375 5143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:30.250638 5143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:30.264070 5148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:30.264109 5151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:30.264601 5149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:30.265370 5143 server_base.cc:1034] running on GCE node
I20260925 03:01:30.265964 5143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:30.267104 5143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:30.268356 5143 hybrid_clock.cc:648] HybridClock initialized: now 1790305290268287 us; error 55 us; skew 500 ppm
I20260925 03:01:30.268675 5143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:30 dist-test-slave-8zvk krb5kdc[5037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305290, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:30.279937 5143 init.cc:377] Logged in from keytab as kudu/127.18.196.193@KRBTEST.COM (short username kudu)
I20260925 03:01:30.281216 5143 webserver.cc:460] Webserver started at http://127.18.196.193:42751/ using document root <none> and password file <none>
I20260925 03:01:30.281850 5143 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:30.281950 5143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:30.282218 5143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:30.285224 5143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "1211d93bffea47929f4ab2b2dcd17a49"
format_stamp: "Formatted at 2026-09-25 03:01:30 on dist-test-slave-8zvk"
server_key: "cfe6a839ec995cd9cacfff810a93c73a"
server_key_iv: "8d268b755eb356f01716b93cefa35178"
server_key_version: "encryptionkey@0"
I20260925 03:01:30.285992 5143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "1211d93bffea47929f4ab2b2dcd17a49"
format_stamp: "Formatted at 2026-09-25 03:01:30 on dist-test-slave-8zvk"
server_key: "cfe6a839ec995cd9cacfff810a93c73a"
server_key_iv: "8d268b755eb356f01716b93cefa35178"
server_key_version: "encryptionkey@0"
I20260925 03:01:30.291023 5143 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260925 03:01:30.294912 5158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:30.296312 5143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 03:01:30.296479 5143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "1211d93bffea47929f4ab2b2dcd17a49"
format_stamp: "Formatted at 2026-09-25 03:01:30 on dist-test-slave-8zvk"
server_key: "cfe6a839ec995cd9cacfff810a93c73a"
server_key_iv: "8d268b755eb356f01716b93cefa35178"
server_key_version: "encryptionkey@0"
I20260925 03:01:30.296672 5143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:30.311292 5143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:30.315017 5143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:30.315284 5143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:30.316251 5143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:01:30.317456 5143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:01:30.317545 5143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:30.317628 5143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:01:30.317692 5143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:30.350243 5143 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:34709
I20260925 03:01:30.350291 5270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:34709 every 8 connection(s)
I20260925 03:01:30.351593 5143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 03:01:30.356837 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 5143
I20260925 03:01:30.357051 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 03:01:30.357654 19219 external_mini_cluster.cc:1428] Setting key e5cc8213c6b376f3e0e5d5ab20b9ed10
Sep 25 03:01:30 dist-test-slave-8zvk krb5kdc[5037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305290, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:01:30.372918 5273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:30.354016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:49493 (local address 127.18.196.254:36937)
0925 03:01:30.354424 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:30.354429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:30.355477 (+ 1048us) server_negotiation.cc:407] Connection header received
0925 03:01:30.357673 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:30.357678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:30.357790 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:30.357893 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:30.359729 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:30.360448 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:30.362981 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:30.363366 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:30.363461 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:30.367494 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:30.367499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:30.367502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:30.367532 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:30.370051 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:30.370725 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:30.370734 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:30.370736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:30.370897 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:30.371305 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:30.371310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:30.371313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:30.371567 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:30.371731 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:30.372370 (+ 639us) server_negotiation.cc:299] Negotiation successful
0925 03:01:30.372575 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":211,"threads_started":1}
I20260925 03:01:30.373788 5274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:30.354577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:36937 (local address 127.18.196.193:49493)
0925 03:01:30.355324 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:30.355360 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:30.357382 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:30.358171 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:30.358182 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:30.358879 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:01:30.359542 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:30.359564 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:30.360606 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:30.360615 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:30.362808 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:30.362827 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:30.363585 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:30.363594 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:30.363923 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:30.364628 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:01:30.364662 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:01:30.367284 (+ 2622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:01:30.370185 (+ 2901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:01:30.370193 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:01:30.370207 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:01:30.370598 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:01:30.371002 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:01:30.371007 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:01:30.371011 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:01:30.371189 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:01:30.371696 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:01:30.371703 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:01:30.372014 (+ 311us) client_negotiation.cc:769] Sending connection context
0925 03:01:30.372250 (+ 236us) client_negotiation.cc:240] Negotiation successful
0925 03:01:30.372715 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":339,"threads_started":1}
I20260925 03:01:30.375360 5271 heartbeater.cc:347] Connected to a master server at 127.18.196.254:36937
I20260925 03:01:30.375660 5271 heartbeater.cc:464] Registering TS with master...
I20260925 03:01:30.376516 5271 heartbeater.cc:511] Master 127.18.196.254:36937 requested a full tablet report, sending...
I20260925 03:01:30.378901 5082 ts_manager.cc:194] Registered new tserver with Master: 1211d93bffea47929f4ab2b2dcd17a49 (127.18.196.193:34709)
I20260925 03:01:30.381052 5082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.193@KRBTEST.COM'} at 127.18.196.193:49493
WARNING: no policy specified for kudu/127.18.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
I20260925 03:01:30.415495 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:36937
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:34303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:01:30.729669 5277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:30.729895 5277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:30.729977 5277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:30.736572 5277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:30.736655 5277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:30.736752 5277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 03:01:30.745412 5277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:34303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:36937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:30.746469 5277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:30.747661 5277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:30.761128 5285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:30.761165 5283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:30.762076 5282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:30.762171 5277 server_base.cc:1034] running on GCE node
I20260925 03:01:30.762888 5277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:30.763895 5277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:30.765132 5277 hybrid_clock.cc:648] HybridClock initialized: now 1790305290765050 us; error 89 us; skew 500 ppm
I20260925 03:01:30.765470 5277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:30 dist-test-slave-8zvk krb5kdc[5037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305290, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:30.768775 5277 init.cc:377] Logged in from keytab as kudu/127.18.196.194@KRBTEST.COM (short username kudu)
I20260925 03:01:30.770409 5277 webserver.cc:460] Webserver started at http://127.18.196.194:37185/ using document root <none> and password file <none>
I20260925 03:01:30.771085 5277 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:30.771207 5277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:30.771514 5277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:30.774037 5277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "053dcae1dad44337856821ac34b49b87"
format_stamp: "Formatted at 2026-09-25 03:01:30 on dist-test-slave-8zvk"
server_key: "02c321e2a5ddbb9da57239d18e594d6d"
server_key_iv: "cf1edb3a77209ab38ebfa8f3c0971caa"
server_key_version: "encryptionkey@0"
I20260925 03:01:30.774712 5277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "053dcae1dad44337856821ac34b49b87"
format_stamp: "Formatted at 2026-09-25 03:01:30 on dist-test-slave-8zvk"
server_key: "02c321e2a5ddbb9da57239d18e594d6d"
server_key_iv: "cf1edb3a77209ab38ebfa8f3c0971caa"
server_key_version: "encryptionkey@0"
I20260925 03:01:30.779537 5277 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 03:01:30.782570 5292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:30.783806 5277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 03:01:30.783979 5277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "053dcae1dad44337856821ac34b49b87"
format_stamp: "Formatted at 2026-09-25 03:01:30 on dist-test-slave-8zvk"
server_key: "02c321e2a5ddbb9da57239d18e594d6d"
server_key_iv: "cf1edb3a77209ab38ebfa8f3c0971caa"
server_key_version: "encryptionkey@0"
I20260925 03:01:30.784173 5277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:30.801090 5277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:30.804775 5277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:30.805068 5277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:30.805990 5277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:01:30.807309 5277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:01:30.807380 5277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:30.807564 5277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:01:30.807612 5277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:30.840025 5277 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:34755
I20260925 03:01:30.840071 5404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:34755 every 8 connection(s)
I20260925 03:01:30.841260 5277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 03:01:30.846164 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 5277
I20260925 03:01:30.846346 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 03:01:30.846788 19219 external_mini_cluster.cc:1428] Setting key 28e90bc88ff791b78f5813fba4736747
Sep 25 03:01:30 dist-test-slave-8zvk krb5kdc[5037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305290, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:01:30.860558 5273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:30.843864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:40861 (local address 127.18.196.254:36937)
0925 03:01:30.844054 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:30.844058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:30.845107 (+ 1049us) server_negotiation.cc:407] Connection header received
0925 03:01:30.847083 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:30.847089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:30.847175 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:30.847283 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:30.848889 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:30.849601 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:30.851114 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:30.851516 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:30.851643 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:30.854913 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:30.854918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:30.854921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:30.854951 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:30.857317 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:30.858221 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:30.858225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:30.858227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:30.858395 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:30.858952 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:30.858956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:30.858958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:30.859207 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:30.859336 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:30.860151 (+ 815us) server_negotiation.cc:299] Negotiation successful
0925 03:01:30.860321 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 03:01:30.861976 5407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:30.844293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:36937 (local address 127.18.196.194:40861)
0925 03:01:30.844965 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:30.845002 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:30.846747 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:30.847498 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:30.847506 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:30.848133 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:01:30.848729 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:30.848745 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:30.849759 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:30.849763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:30.850984 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:30.850995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:30.851726 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:30.851733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:30.851977 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:30.852638 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:01:30.852680 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:01:30.854673 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:01:30.857464 (+ 2791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:01:30.857474 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:01:30.857491 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:01:30.858102 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:01:30.858529 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:01:30.858537 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:01:30.858543 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:01:30.858792 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:01:30.859346 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:01:30.859356 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:01:30.859766 (+ 410us) client_negotiation.cc:769] Sending connection context
0925 03:01:30.860073 (+ 307us) client_negotiation.cc:240] Negotiation successful
0925 03:01:30.860580 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":259,"threads_started":1}
I20260925 03:01:30.863345 5405 heartbeater.cc:347] Connected to a master server at 127.18.196.254:36937
I20260925 03:01:30.863641 5405 heartbeater.cc:464] Registering TS with master...
I20260925 03:01:30.864305 5405 heartbeater.cc:511] Master 127.18.196.254:36937 requested a full tablet report, sending...
I20260925 03:01:30.865715 5082 ts_manager.cc:194] Registered new tserver with Master: 053dcae1dad44337856821ac34b49b87 (127.18.196.194:34755)
I20260925 03:01:30.866508 5082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.194@KRBTEST.COM'} at 127.18.196.194:40861
WARNING: no policy specified for kudu/127.18.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
I20260925 03:01:30.902053 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:36937
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:34303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:01:31.229818 5410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:31.230093 5410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:31.230247 5410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:31.237530 5410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:31.237636 5410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:31.237847 5410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 03:01:31.247500 5410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:34303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:36937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:31.248703 5410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:31.249986 5410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:31.262761 5415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:31.262737 5416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:31.262943 5418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:31.263571 5410 server_base.cc:1034] running on GCE node
I20260925 03:01:31.264212 5410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:31.265137 5410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:31.266328 5410 hybrid_clock.cc:648] HybridClock initialized: now 1790305291266291 us; error 33 us; skew 500 ppm
I20260925 03:01:31.266628 5410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:31 dist-test-slave-8zvk krb5kdc[5037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305291, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:31.269629 5410 init.cc:377] Logged in from keytab as kudu/127.18.196.195@KRBTEST.COM (short username kudu)
I20260925 03:01:31.271072 5410 webserver.cc:460] Webserver started at http://127.18.196.195:37827/ using document root <none> and password file <none>
I20260925 03:01:31.271725 5410 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:31.271868 5410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:31.272192 5410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:31.275002 5410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "1c368510d51b4a15bfe9c9a0646d3011"
format_stamp: "Formatted at 2026-09-25 03:01:31 on dist-test-slave-8zvk"
server_key: "10deecb8b928a4506c12195a394c9d24"
server_key_iv: "37d0df00e34b7437a3ea23d596f12406"
server_key_version: "encryptionkey@0"
I20260925 03:01:31.275748 5410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "1c368510d51b4a15bfe9c9a0646d3011"
format_stamp: "Formatted at 2026-09-25 03:01:31 on dist-test-slave-8zvk"
server_key: "10deecb8b928a4506c12195a394c9d24"
server_key_iv: "37d0df00e34b7437a3ea23d596f12406"
server_key_version: "encryptionkey@0"
I20260925 03:01:31.280718 5410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 03:01:31.284003 5425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:31.285218 5410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 03:01:31.285389 5410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "1c368510d51b4a15bfe9c9a0646d3011"
format_stamp: "Formatted at 2026-09-25 03:01:31 on dist-test-slave-8zvk"
server_key: "10deecb8b928a4506c12195a394c9d24"
server_key_iv: "37d0df00e34b7437a3ea23d596f12406"
server_key_version: "encryptionkey@0"
I20260925 03:01:31.285578 5410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:31.319600 5410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:31.323450 5410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:31.323738 5410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:31.324693 5410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:01:31.326061 5410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:01:31.326164 5410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:31.326323 5410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:01:31.326396 5410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:31.359745 5410 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:42529
I20260925 03:01:31.359802 5537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:42529 every 8 connection(s)
I20260925 03:01:31.360956 5410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 03:01:31.365974 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 5410
I20260925 03:01:31.366161 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 03:01:31.366696 19219 external_mini_cluster.cc:1428] Setting key 3af4c69293028e7a463833701366b70e
Sep 25 03:01:31 dist-test-slave-8zvk krb5kdc[5037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305291, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:01:31.380138 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:31.363413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:37933 (local address 127.18.196.254:36937)
0925 03:01:31.363890 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:31.363895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:31.364694 (+ 799us) server_negotiation.cc:407] Connection header received
0925 03:01:31.366989 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:31.366994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:31.367066 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:31.367182 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:31.368989 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.370001 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.371545 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.371915 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.372003 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:31.375079 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:31.375083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:31.375086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:31.375118 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:31.377509 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:31.378154 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:31.378158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:31.378160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:31.378278 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:31.378624 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:31.378627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:31.378629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:31.378950 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:31.379063 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:31.379626 (+ 563us) server_negotiation.cc:299] Negotiation successful
0925 03:01:31.379775 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":259,"threads_started":1}
I20260925 03:01:31.381217 5541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:31.363905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:36937 (local address 127.18.196.195:37933)
0925 03:01:31.364539 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:31.364582 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:31.366662 (+ 2080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:31.367390 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:31.367399 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:31.368129 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:01:31.368830 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:31.368848 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.370131 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.370135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:31.371426 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:31.371439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.372083 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.372087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:31.372256 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:31.372834 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:01:31.372859 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:01:31.374899 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:01:31.377630 (+ 2731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:01:31.377636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:01:31.377646 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:01:31.378040 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:01:31.378367 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:01:31.378370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:01:31.378372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:01:31.378528 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:01:31.379051 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:01:31.379056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:01:31.379320 (+ 264us) client_negotiation.cc:769] Sending connection context
0925 03:01:31.379527 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 03:01:31.379973 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":258,"threads_started":1}
I20260925 03:01:31.382469 5538 heartbeater.cc:347] Connected to a master server at 127.18.196.254:36937
I20260925 03:01:31.382755 5538 heartbeater.cc:464] Registering TS with master...
I20260925 03:01:31.383452 5538 heartbeater.cc:511] Master 127.18.196.254:36937 requested a full tablet report, sending...
I20260925 03:01:31.384719 5271 heartbeater.cc:503] Master 127.18.196.254:36937 was elected leader, sending a full tablet report...
I20260925 03:01:31.384882 5082 ts_manager.cc:194] Registered new tserver with Master: 1c368510d51b4a15bfe9c9a0646d3011 (127.18.196.195:42529)
I20260925 03:01:31.385766 5082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.195@KRBTEST.COM'} at 127.18.196.195:37933
I20260925 03:01:31.394170 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 03:01:31.411170 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:31.398547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.18.196.254:36937)
0925 03:01:31.398729 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:31.398733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:31.399195 (+ 462us) server_negotiation.cc:407] Connection header received
0925 03:01:31.399329 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:31.399333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:31.399398 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:31.399483 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:31.400905 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.401780 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.403105 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.403526 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.403620 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:31.406048 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:31.406053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:31.406056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:31.406086 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:31.408711 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:31.409372 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:31.409376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:31.409378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:31.409487 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:31.409830 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:31.409833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:31.409835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:31.410142 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:31.410340 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:31.410631 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 03:01:31.410876 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 03:01:31.415020 5082 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 03:01:31.421636 5082 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 03:01:31.460398 5551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:31.432818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:42529 (local address 127.0.0.1:60698)
0925 03:01:31.434012 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:31.434062 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:31.434271 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:31.452550 (+ 18279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:31.452555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:31.452599 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:31.453404 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:31.453429 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.454897 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.454902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:31.457574 (+ 2672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:31.457589 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.459459 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.459465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:31.459665 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:31.459723 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 03:01:31.459846 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 03:01:31.460007 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":206,"spinlock_wait_cycles":1408,"thread_start_us":736,"threads_started":1}
I20260925 03:01:31.460443 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:31.433101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60698 (local address 127.18.196.195:42529)
0925 03:01:31.451915 (+ 18814us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:31.451923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:31.451956 (+ 33us) server_negotiation.cc:407] Connection header received
0925 03:01:31.452063 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:31.452070 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:31.452314 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:31.453195 (+ 881us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:31.453659 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.454728 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.457777 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.459218 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.459595 (+ 377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:31.459757 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:31.459912 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 03:01:31.460076 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18637,"thread_start_us":18501,"threads_started":1}
I20260925 03:01:31.460891 5555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:31.433654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57762 (local address 127.18.196.194:34755)
0925 03:01:31.451060 (+ 17406us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:31.451073 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:31.451842 (+ 769us) server_negotiation.cc:407] Connection header received
0925 03:01:31.451974 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:31.451985 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:31.452282 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:31.452499 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:31.453538 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.455206 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.458040 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.459506 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.459680 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:31.459814 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:31.460376 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 03:01:31.460555 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17196,"thread_start_us":17029,"threads_started":1}
I20260925 03:01:31.461294 5554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:31.433097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:34755 (local address 127.0.0.1:57762)
0925 03:01:31.451582 (+ 18485us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:31.451605 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:31.451742 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:31.452532 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:31.452539 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:31.452570 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:31.453321 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:31.453340 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.455354 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.455360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:31.457859 (+ 2499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:31.457873 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.459970 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.459977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:31.460152 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:31.460194 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 03:01:31.460296 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 03:01:31.460423 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18320,"mutex_wait_us":63,"thread_start_us":18136,"threads_started":1}
I20260925 03:01:31.461545 5556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:31.434406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:34709 (local address 127.0.0.1:45872)
0925 03:01:31.452219 (+ 17813us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:31.452247 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:31.452357 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:31.452974 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:31.452981 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:31.453012 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:31.453559 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:31.453591 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.455419 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.455426 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:31.457788 (+ 2362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:31.457805 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.459305 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.459311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:31.460840 (+ 1529us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:31.460884 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 03:01:31.460992 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 03:01:31.461121 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17687,"mutex_wait_us":88,"thread_start_us":17725,"threads_started":1}
I20260925 03:01:31.461928 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:31.434999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45872 (local address 127.18.196.193:34709)
0925 03:01:31.450220 (+ 15221us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:31.450233 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:31.452355 (+ 2122us) server_negotiation.cc:407] Connection header received
0925 03:01:31.452529 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:31.452536 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:31.452763 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:31.452977 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:31.453755 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.455269 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.457977 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.459108 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.459312 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:31.459501 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:31.461397 (+ 1896us) server_negotiation.cc:299] Negotiation successful
0925 03:01:31.461575 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14986,"thread_start_us":15231,"threads_started":1}
I20260925 03:01:31.464264 5473 tablet_service.cc:1467] Processing CreateTablet for tablet 9d0343501dc147a9aad3351da24a87ea (DEFAULT_TABLE table=test-table [id=295b6d8639a04dc883c0a4a4bea80267]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 03:01:31.464895 5340 tablet_service.cc:1467] Processing CreateTablet for tablet 9d0343501dc147a9aad3351da24a87ea (DEFAULT_TABLE table=test-table [id=295b6d8639a04dc883c0a4a4bea80267]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 03:01:31.465637 5473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d0343501dc147a9aad3351da24a87ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:31.465734 5206 tablet_service.cc:1467] Processing CreateTablet for tablet 9d0343501dc147a9aad3351da24a87ea (DEFAULT_TABLE table=test-table [id=295b6d8639a04dc883c0a4a4bea80267]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 03:01:31.466226 5340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d0343501dc147a9aad3351da24a87ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:31.467068 5206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d0343501dc147a9aad3351da24a87ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:31.476260 5558 tablet_bootstrap.cc:492] T 9d0343501dc147a9aad3351da24a87ea P 1c368510d51b4a15bfe9c9a0646d3011: Bootstrap starting.
I20260925 03:01:31.479144 5558 tablet_bootstrap.cc:654] T 9d0343501dc147a9aad3351da24a87ea P 1c368510d51b4a15bfe9c9a0646d3011: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:31.480568 5558 log.cc:824] T 9d0343501dc147a9aad3351da24a87ea P 1c368510d51b4a15bfe9c9a0646d3011: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:31.483520 5560 tablet_bootstrap.cc:492] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49: Bootstrap starting.
I20260925 03:01:31.483574 5558 tablet_bootstrap.cc:492] T 9d0343501dc147a9aad3351da24a87ea P 1c368510d51b4a15bfe9c9a0646d3011: No bootstrap required, opened a new log
I20260925 03:01:31.483850 5558 ts_tablet_manager.cc:1397] T 9d0343501dc147a9aad3351da24a87ea P 1c368510d51b4a15bfe9c9a0646d3011: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 03:01:31.486228 5560 tablet_bootstrap.cc:654] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:31.486907 5559 tablet_bootstrap.cc:492] T 9d0343501dc147a9aad3351da24a87ea P 053dcae1dad44337856821ac34b49b87: Bootstrap starting.
I20260925 03:01:31.487353 5560 log.cc:824] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:31.490165 5560 tablet_bootstrap.cc:492] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49: No bootstrap required, opened a new log
I20260925 03:01:31.490429 5560 ts_tablet_manager.cc:1397] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 03:01:31.490947 5559 tablet_bootstrap.cc:654] T 9d0343501dc147a9aad3351da24a87ea P 053dcae1dad44337856821ac34b49b87: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:31.492451 5559 log.cc:824] T 9d0343501dc147a9aad3351da24a87ea P 053dcae1dad44337856821ac34b49b87: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:31.493189 5558 raft_consensus.cc:348] T 9d0343501dc147a9aad3351da24a87ea P 1c368510d51b4a15bfe9c9a0646d3011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c368510d51b4a15bfe9c9a0646d3011" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 42529 } } peers { permanent_uuid: "053dcae1dad44337856821ac34b49b87" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 34755 } } peers { permanent_uuid: "1211d93bffea47929f4ab2b2dcd17a49" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 34709 } }
I20260925 03:01:31.493608 5558 raft_consensus.cc:374] T 9d0343501dc147a9aad3351da24a87ea P 1c368510d51b4a15bfe9c9a0646d3011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:31.493713 5558 raft_consensus.cc:729] T 9d0343501dc147a9aad3351da24a87ea P 1c368510d51b4a15bfe9c9a0646d3011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c368510d51b4a15bfe9c9a0646d3011, State: Initialized, Role: FOLLOWER
I20260925 03:01:31.494350 5558 consensus_queue.cc:260] T 9d0343501dc147a9aad3351da24a87ea P 1c368510d51b4a15bfe9c9a0646d3011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c368510d51b4a15bfe9c9a0646d3011" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 42529 } } peers { permanent_uuid: "053dcae1dad44337856821ac34b49b87" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 34755 } } peers { permanent_uuid: "1211d93bffea47929f4ab2b2dcd17a49" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 34709 } }
I20260925 03:01:31.495510 5538 heartbeater.cc:503] Master 127.18.196.254:36937 was elected leader, sending a full tablet report...
I20260925 03:01:31.495730 5559 tablet_bootstrap.cc:492] T 9d0343501dc147a9aad3351da24a87ea P 053dcae1dad44337856821ac34b49b87: No bootstrap required, opened a new log
I20260925 03:01:31.496006 5559 ts_tablet_manager.cc:1397] T 9d0343501dc147a9aad3351da24a87ea P 053dcae1dad44337856821ac34b49b87: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.006s
I20260925 03:01:31.496244 5558 ts_tablet_manager.cc:1428] T 9d0343501dc147a9aad3351da24a87ea P 1c368510d51b4a15bfe9c9a0646d3011: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260925 03:01:31.502193 5560 raft_consensus.cc:348] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c368510d51b4a15bfe9c9a0646d3011" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 42529 } } peers { permanent_uuid: "053dcae1dad44337856821ac34b49b87" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 34755 } } peers { permanent_uuid: "1211d93bffea47929f4ab2b2dcd17a49" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 34709 } }
I20260925 03:01:31.502642 5560 raft_consensus.cc:374] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:31.502753 5560 raft_consensus.cc:729] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1211d93bffea47929f4ab2b2dcd17a49, State: Initialized, Role: FOLLOWER
I20260925 03:01:31.503377 5560 consensus_queue.cc:260] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c368510d51b4a15bfe9c9a0646d3011" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 42529 } } peers { permanent_uuid: "053dcae1dad44337856821ac34b49b87" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 34755 } } peers { permanent_uuid: "1211d93bffea47929f4ab2b2dcd17a49" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 34709 } }
I20260925 03:01:31.505095 5560 ts_tablet_manager.cc:1428] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260925 03:01:31.506176 5559 raft_consensus.cc:348] T 9d0343501dc147a9aad3351da24a87ea P 053dcae1dad44337856821ac34b49b87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c368510d51b4a15bfe9c9a0646d3011" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 42529 } } peers { permanent_uuid: "053dcae1dad44337856821ac34b49b87" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 34755 } } peers { permanent_uuid: "1211d93bffea47929f4ab2b2dcd17a49" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 34709 } }
I20260925 03:01:31.506548 5559 raft_consensus.cc:374] T 9d0343501dc147a9aad3351da24a87ea P 053dcae1dad44337856821ac34b49b87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:31.506657 5559 raft_consensus.cc:729] T 9d0343501dc147a9aad3351da24a87ea P 053dcae1dad44337856821ac34b49b87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 053dcae1dad44337856821ac34b49b87, State: Initialized, Role: FOLLOWER
I20260925 03:01:31.507215 5559 consensus_queue.cc:260] T 9d0343501dc147a9aad3351da24a87ea P 053dcae1dad44337856821ac34b49b87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c368510d51b4a15bfe9c9a0646d3011" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 42529 } } peers { permanent_uuid: "053dcae1dad44337856821ac34b49b87" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 34755 } } peers { permanent_uuid: "1211d93bffea47929f4ab2b2dcd17a49" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 34709 } }
I20260925 03:01:31.508276 5405 heartbeater.cc:503] Master 127.18.196.254:36937 was elected leader, sending a full tablet report...
I20260925 03:01:31.508765 5559 ts_tablet_manager.cc:1428] T 9d0343501dc147a9aad3351da24a87ea P 053dcae1dad44337856821ac34b49b87: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
W20260925 03:01:31.593375 5406 tablet.cc:2369] T 9d0343501dc147a9aad3351da24a87ea P 053dcae1dad44337856821ac34b49b87: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 03:01:31.603961 5272 tablet.cc:2369] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 03:01:31.612586 5539 tablet.cc:2369] T 9d0343501dc147a9aad3351da24a87ea P 1c368510d51b4a15bfe9c9a0646d3011: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 03:01:31.804275 5565 raft_consensus.cc:482] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:01:31.804632 5565 raft_consensus.cc:504] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c368510d51b4a15bfe9c9a0646d3011" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 42529 } } peers { permanent_uuid: "053dcae1dad44337856821ac34b49b87" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 34755 } } peers { permanent_uuid: "1211d93bffea47929f4ab2b2dcd17a49" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 34709 } }
I20260925 03:01:31.806619 5565 leader_election.cc:288] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c368510d51b4a15bfe9c9a0646d3011 (127.18.196.195:42529), 053dcae1dad44337856821ac34b49b87 (127.18.196.194:34755)
I20260925 03:01:31.814339 5555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:31.807448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:48091 (local address 127.18.196.194:34755)
0925 03:01:31.807661 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:31.807667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:31.807689 (+ 22us) server_negotiation.cc:407] Connection header received
0925 03:01:31.807993 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:31.808001 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:31.808105 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:31.808204 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:31.809034 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.810074 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.812643 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.813378 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.813548 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:31.813627 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:31.813921 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 03:01:31.814046 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 03:01:31.814385 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:31.806992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:35351 (local address 127.18.196.195:42529)
0925 03:01:31.807231 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:31.807240 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:31.807974 (+ 734us) server_negotiation.cc:407] Connection header received
0925 03:01:31.808112 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:31.808116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:31.808197 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:31.808276 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:31.809034 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.810089 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.812682 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.813423 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.813602 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:31.813680 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:31.813971 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 03:01:31.814101 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 03:01:31.814543 5568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:31.806911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:42529 (local address 127.18.196.193:35351)
0925 03:01:31.807873 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:31.807891 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:31.807998 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:31.808312 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:31.808318 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:31.808345 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:31.808894 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:31.808903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.810223 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.810229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:31.812525 (+ 2296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:31.812540 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.813651 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.813657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:31.813819 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:31.813841 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 03:01:31.813942 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 03:01:31.814061 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":151,"thread_start_us":634,"threads_started":1}
I20260925 03:01:31.814885 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:31.806896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:34755 (local address 127.18.196.193:48091)
0925 03:01:31.807600 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:31.807626 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:31.807808 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:31.808229 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:31.808232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:31.808276 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:31.808854 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:31.808873 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.810223 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.810229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:31.812480 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:31.812496 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.813601 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.813606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:31.813759 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:31.813782 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 03:01:31.813887 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 03:01:31.814014 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":227,"thread_start_us":359,"threads_started":1}
I20260925 03:01:31.815642 5493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0343501dc147a9aad3351da24a87ea" candidate_uuid: "1211d93bffea47929f4ab2b2dcd17a49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c368510d51b4a15bfe9c9a0646d3011" is_pre_election: true
I20260925 03:01:31.815905 5360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0343501dc147a9aad3351da24a87ea" candidate_uuid: "1211d93bffea47929f4ab2b2dcd17a49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "053dcae1dad44337856821ac34b49b87" is_pre_election: true
I20260925 03:01:31.816102 5493 raft_consensus.cc:2454] T 9d0343501dc147a9aad3351da24a87ea P 1c368510d51b4a15bfe9c9a0646d3011 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1211d93bffea47929f4ab2b2dcd17a49 in term 0.
I20260925 03:01:31.816442 5360 raft_consensus.cc:2454] T 9d0343501dc147a9aad3351da24a87ea P 053dcae1dad44337856821ac34b49b87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1211d93bffea47929f4ab2b2dcd17a49 in term 0.
I20260925 03:01:31.816777 5159 leader_election.cc:302] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1211d93bffea47929f4ab2b2dcd17a49, 1c368510d51b4a15bfe9c9a0646d3011; no voters:
I20260925 03:01:31.817242 5565 raft_consensus.cc:2781] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:01:31.817406 5565 raft_consensus.cc:482] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:01:31.817503 5565 raft_consensus.cc:3037] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:31.819931 5565 raft_consensus.cc:504] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c368510d51b4a15bfe9c9a0646d3011" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 42529 } } peers { permanent_uuid: "053dcae1dad44337856821ac34b49b87" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 34755 } } peers { permanent_uuid: "1211d93bffea47929f4ab2b2dcd17a49" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 34709 } }
I20260925 03:01:31.820557 5565 leader_election.cc:288] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49 [CANDIDATE]: Term 1 election: Requested vote from peers 1c368510d51b4a15bfe9c9a0646d3011 (127.18.196.195:42529), 053dcae1dad44337856821ac34b49b87 (127.18.196.194:34755)
I20260925 03:01:31.821048 5360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0343501dc147a9aad3351da24a87ea" candidate_uuid: "1211d93bffea47929f4ab2b2dcd17a49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "053dcae1dad44337856821ac34b49b87"
I20260925 03:01:31.821048 5493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0343501dc147a9aad3351da24a87ea" candidate_uuid: "1211d93bffea47929f4ab2b2dcd17a49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c368510d51b4a15bfe9c9a0646d3011"
I20260925 03:01:31.821257 5360 raft_consensus.cc:3037] T 9d0343501dc147a9aad3351da24a87ea P 053dcae1dad44337856821ac34b49b87 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:31.821257 5493 raft_consensus.cc:3037] T 9d0343501dc147a9aad3351da24a87ea P 1c368510d51b4a15bfe9c9a0646d3011 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:31.824484 5360 raft_consensus.cc:2454] T 9d0343501dc147a9aad3351da24a87ea P 053dcae1dad44337856821ac34b49b87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1211d93bffea47929f4ab2b2dcd17a49 in term 1.
I20260925 03:01:31.824488 5493 raft_consensus.cc:2454] T 9d0343501dc147a9aad3351da24a87ea P 1c368510d51b4a15bfe9c9a0646d3011 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1211d93bffea47929f4ab2b2dcd17a49 in term 1.
I20260925 03:01:31.824863 5159 leader_election.cc:302] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1211d93bffea47929f4ab2b2dcd17a49, 1c368510d51b4a15bfe9c9a0646d3011; no voters:
I20260925 03:01:31.825179 5565 raft_consensus.cc:2781] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:01:31.825628 5565 raft_consensus.cc:686] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49 [term 1 LEADER]: Becoming Leader. State: Replica: 1211d93bffea47929f4ab2b2dcd17a49, State: Running, Role: LEADER
I20260925 03:01:31.826171 5565 consensus_queue.cc:237] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c368510d51b4a15bfe9c9a0646d3011" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 42529 } } peers { permanent_uuid: "053dcae1dad44337856821ac34b49b87" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 34755 } } peers { permanent_uuid: "1211d93bffea47929f4ab2b2dcd17a49" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 34709 } }
I20260925 03:01:31.830641 5082 catalog_manager.cc:5495] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1211d93bffea47929f4ab2b2dcd17a49 (127.18.196.193). New cstate: current_term: 1 leader_uuid: "1211d93bffea47929f4ab2b2dcd17a49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c368510d51b4a15bfe9c9a0646d3011" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 42529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "053dcae1dad44337856821ac34b49b87" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1211d93bffea47929f4ab2b2dcd17a49" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 34709 } health_report { overall_health: HEALTHY } } }
I20260925 03:01:31.906711 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:31.900938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45884 (local address 127.18.196.193:34709)
0925 03:01:31.901151 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:31.901157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:31.901233 (+ 76us) server_negotiation.cc:407] Connection header received
0925 03:01:31.901502 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:31.901506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:31.901605 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:31.901764 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:01:31.902198 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.902976 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.904403 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.904814 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.904969 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:31.905606 (+ 637us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:01:31.906096 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:01:31.906306 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:31.906419 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 03:01:31.906464 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 03:01:31.915002 5493 raft_consensus.cc:1261] T 9d0343501dc147a9aad3351da24a87ea P 1c368510d51b4a15bfe9c9a0646d3011 [term 1 FOLLOWER]: Refusing update from remote peer 1211d93bffea47929f4ab2b2dcd17a49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:01:31.915006 5360 raft_consensus.cc:1261] T 9d0343501dc147a9aad3351da24a87ea P 053dcae1dad44337856821ac34b49b87 [term 1 FOLLOWER]: Refusing update from remote peer 1211d93bffea47929f4ab2b2dcd17a49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:01:31.915967 5569 consensus_queue.cc:1035] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "053dcae1dad44337856821ac34b49b87" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:01:31.916356 5565 consensus_queue.cc:1035] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c368510d51b4a15bfe9c9a0646d3011" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 42529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:01:31.927790 5572 mvcc.cc:205] Tried to move back new op lower bound from 7333090475674124288 to 7333090475324309504. Current Snapshot: MvccSnapshot[applied={T|T < 7333090475674124288}]
I20260925 03:01:31.931504 5574 mvcc.cc:205] Tried to move back new op lower bound from 7333090475674124288 to 7333090475324309504. Current Snapshot: MvccSnapshot[applied={T|T < 7333090475674124288}]
I20260925 03:01:31.945838 5082 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 03:01:31.946096 5082 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 03:01:31.949565 5082 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2199ae0a5d3403a8c3760c75d6068aa: Sending DeleteTablet for 3 replicas of tablet 9d0343501dc147a9aad3351da24a87ea
I20260925 03:01:31.950692 5473 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d0343501dc147a9aad3351da24a87ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 03:01:31 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:60698
I20260925 03:01:31.950899 5206 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d0343501dc147a9aad3351da24a87ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 03:01:31 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:45872
I20260925 03:01:31.951356 5340 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d0343501dc147a9aad3351da24a87ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 03:01:31 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:57762
I20260925 03:01:31.951998 5582 tablet_replica.cc:330] T 9d0343501dc147a9aad3351da24a87ea P 1c368510d51b4a15bfe9c9a0646d3011: stopping tablet replica
I20260925 03:01:31.952553 5582 raft_consensus.cc:2229] T 9d0343501dc147a9aad3351da24a87ea P 1c368510d51b4a15bfe9c9a0646d3011 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:01:31.953166 5582 raft_consensus.cc:2258] T 9d0343501dc147a9aad3351da24a87ea P 1c368510d51b4a15bfe9c9a0646d3011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:01:31.953763 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 5143
I20260925 03:01:31.954102 5583 tablet_replica.cc:330] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49: stopping tablet replica
I20260925 03:01:31.954221 5584 tablet_replica.cc:330] T 9d0343501dc147a9aad3351da24a87ea P 053dcae1dad44337856821ac34b49b87: stopping tablet replica
I20260925 03:01:31.954571 5584 raft_consensus.cc:2229] T 9d0343501dc147a9aad3351da24a87ea P 053dcae1dad44337856821ac34b49b87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:01:31.954588 5583 raft_consensus.cc:2229] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49 [term 1 LEADER]: Raft consensus shutting down.
I20260925 03:01:31.955076 5584 raft_consensus.cc:2258] T 9d0343501dc147a9aad3351da24a87ea P 053dcae1dad44337856821ac34b49b87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:01:31.955428 5583 raft_consensus.cc:2258] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:01:31.955842 5582 ts_tablet_manager.cc:1905] T 9d0343501dc147a9aad3351da24a87ea P 1c368510d51b4a15bfe9c9a0646d3011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 03:01:31.957376 5583 ts_tablet_manager.cc:1905] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 03:01:31.957376 5584 ts_tablet_manager.cc:1905] T 9d0343501dc147a9aad3351da24a87ea P 053dcae1dad44337856821ac34b49b87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 03:01:31.964438 5583 ts_tablet_manager.cc:1918] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 03:01:31.964532 5584 ts_tablet_manager.cc:1918] T 9d0343501dc147a9aad3351da24a87ea P 053dcae1dad44337856821ac34b49b87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 03:01:31.964709 5583 log.cc:1196] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-0/wal/wals/9d0343501dc147a9aad3351da24a87ea
I20260925 03:01:31.964797 5582 ts_tablet_manager.cc:1918] T 9d0343501dc147a9aad3351da24a87ea P 1c368510d51b4a15bfe9c9a0646d3011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 03:01:31.964967 5582 log.cc:1196] T 9d0343501dc147a9aad3351da24a87ea P 1c368510d51b4a15bfe9c9a0646d3011: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/9d0343501dc147a9aad3351da24a87ea
I20260925 03:01:31.964675 5584 log.cc:1196] T 9d0343501dc147a9aad3351da24a87ea P 053dcae1dad44337856821ac34b49b87: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/9d0343501dc147a9aad3351da24a87ea
I20260925 03:01:31.965517 5582 ts_tablet_manager.cc:1939] T 9d0343501dc147a9aad3351da24a87ea P 1c368510d51b4a15bfe9c9a0646d3011: Deleting consensus metadata
I20260925 03:01:31.965513 5583 ts_tablet_manager.cc:1939] T 9d0343501dc147a9aad3351da24a87ea P 1211d93bffea47929f4ab2b2dcd17a49: Deleting consensus metadata
I20260925 03:01:31.966024 5584 ts_tablet_manager.cc:1939] T 9d0343501dc147a9aad3351da24a87ea P 053dcae1dad44337856821ac34b49b87: Deleting consensus metadata
I20260925 03:01:31.967290 5066 catalog_manager.cc:4841] TS 1211d93bffea47929f4ab2b2dcd17a49 (127.18.196.193:34709): tablet 9d0343501dc147a9aad3351da24a87ea (table test-table [id=295b6d8639a04dc883c0a4a4bea80267]) successfully deleted
I20260925 03:01:31.967635 5066 catalog_manager.cc:4841] TS 1c368510d51b4a15bfe9c9a0646d3011 (127.18.196.195:42529): tablet 9d0343501dc147a9aad3351da24a87ea (table test-table [id=295b6d8639a04dc883c0a4a4bea80267]) successfully deleted
I20260925 03:01:31.967700 5067 catalog_manager.cc:4841] TS 053dcae1dad44337856821ac34b49b87 (127.18.196.194:34755): tablet 9d0343501dc147a9aad3351da24a87ea (table test-table [id=295b6d8639a04dc883c0a4a4bea80267]) successfully deleted
Sep 25 03:01:31 dist-test-slave-8zvk krb5kdc[5037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.193@KRBTEST.COM
I20260925 03:01:31.984422 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:31.956148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45888 (local address 127.18.196.193:34709)
0925 03:01:31.956341 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:31.956347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:31.968280 (+ 11933us) server_negotiation.cc:407] Connection header received
0925 03:01:31.968461 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:31.968464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:31.968525 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:31.968614 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:31.970275 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.971214 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.973143 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:31.973611 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:31.973702 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:31.978492 (+ 4790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:31.978506 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:31.978519 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:31.978549 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:31.981569 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:31.982302 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:31.982309 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:31.982312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:31.982449 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:31.982878 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:31.982883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:31.982887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:31.983233 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:31.983560 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:31.983842 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 03:01:31.984076 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 03:01:31.985106 5266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:32.081310 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 5143
I20260925 03:01:32.100450 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 5277
Sep 25 03:01:32 dist-test-slave-8zvk krb5kdc[5037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.194@KRBTEST.COM
I20260925 03:01:32.115828 5555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:32.101235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57778 (local address 127.18.196.194:34755)
0925 03:01:32.101412 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:32.101416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:32.101435 (+ 19us) server_negotiation.cc:407] Connection header received
0925 03:01:32.101579 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:32.101584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:32.101655 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:32.101737 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:32.102977 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:32.103678 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:32.105014 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:32.105392 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:32.105487 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:32.109844 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:32.109857 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:32.109872 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:32.109903 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:32.112921 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:32.113490 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:32.113496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:32.113498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:32.113704 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:32.114195 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:32.114202 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:32.114205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:32.114738 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:32.115069 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:32.115425 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 03:01:32.115574 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 03:01:32.116534 5400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:32.203449 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 5277
I20260925 03:01:32.223043 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 5410
Sep 25 03:01:32 dist-test-slave-8zvk krb5kdc[5037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.195@KRBTEST.COM
I20260925 03:01:32.237012 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:32.223797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60708 (local address 127.18.196.195:42529)
0925 03:01:32.223964 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:32.223969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:32.223986 (+ 17us) server_negotiation.cc:407] Connection header received
0925 03:01:32.224127 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:32.224131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:32.224194 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:32.224284 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:32.225499 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:32.226202 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:32.227492 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:32.227828 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:32.227935 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:32.231993 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:32.232007 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:32.232022 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:32.232052 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:32.234721 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:32.235434 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:32.235442 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:32.235444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:32.235571 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:32.235874 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:32.235876 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:32.235878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:32.236193 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:32.236411 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:32.236616 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 03:01:32.236761 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 03:01:32.237638 5533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:32.323998 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 5410
I20260925 03:01:32.340610 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 5050
I20260925 03:01:32.341696 5112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:32.446964 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 5050
2026-09-25T03:01:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4694 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T03:01:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:01:32Z Disabled control of system clock
I20260925 03:01:32.523910 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:45949
--webserver_interface=127.18.196.254
--webserver_port=0
--builtin_ntp_servers=127.18.196.212:42013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:45949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 03:01:32.842614 5593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:32.843019 5593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:32.843104 5593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:32.849915 5593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:01:32.849998 5593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:32.850035 5593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:32.850082 5593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:01:32.850116 5593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:01:32.858758 5593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:42013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:45949
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:45949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:32.859920 5593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:32.861591 5593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:32.873236 5598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:32.873353 5601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:32.873342 5599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:32.873979 5593 server_base.cc:1034] running on GCE node
I20260925 03:01:32.874758 5593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:32.876152 5593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:32.877385 5593 hybrid_clock.cc:648] HybridClock initialized: now 1790305292877353 us; error 40 us; skew 500 ppm
I20260925 03:01:32.877738 5593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:01:32.879949 5593 webserver.cc:460] Webserver started at http://127.18.196.254:34273/ using document root <none> and password file <none>
I20260925 03:01:32.880599 5593 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:32.880726 5593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:32.881038 5593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:32.884003 5593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "b0806e55a1f14e339555e76c52c1c656"
format_stamp: "Formatted at 2026-09-25 03:01:32 on dist-test-slave-8zvk"
server_key: "92d8d7630cbaddefda493cc885d73e13"
server_key_iv: "0ba8b752119223f55d8e02fc1ce8df7f"
server_key_version: "encryptionkey@0"
I20260925 03:01:32.884801 5593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "b0806e55a1f14e339555e76c52c1c656"
format_stamp: "Formatted at 2026-09-25 03:01:32 on dist-test-slave-8zvk"
server_key: "92d8d7630cbaddefda493cc885d73e13"
server_key_iv: "0ba8b752119223f55d8e02fc1ce8df7f"
server_key_version: "encryptionkey@0"
I20260925 03:01:32.889710 5593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260925 03:01:32.893159 5607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:32.894448 5593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 03:01:32.894637 5593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "b0806e55a1f14e339555e76c52c1c656"
format_stamp: "Formatted at 2026-09-25 03:01:32 on dist-test-slave-8zvk"
server_key: "92d8d7630cbaddefda493cc885d73e13"
server_key_iv: "0ba8b752119223f55d8e02fc1ce8df7f"
server_key_version: "encryptionkey@0"
I20260925 03:01:32.894896 5593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:32.914413 5593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:32.915293 5593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:32.915573 5593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:32.933671 5593 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:45949
I20260925 03:01:32.933697 5659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:45949 every 8 connection(s)
I20260925 03:01:32.935070 5593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:01:32.938627 5660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:32.944444 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 5593
I20260925 03:01:32.944630 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:01:32.945081 19219 external_mini_cluster.cc:1428] Setting key b8f2fd492690f7c5f06316e2affd1439
I20260925 03:01:32.949607 5660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0806e55a1f14e339555e76c52c1c656: Bootstrap starting.
I20260925 03:01:32.952971 5660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0806e55a1f14e339555e76c52c1c656: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:32.953891 5660 log.cc:824] T 00000000000000000000000000000000 P b0806e55a1f14e339555e76c52c1c656: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:32.956815 5660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0806e55a1f14e339555e76c52c1c656: No bootstrap required, opened a new log
I20260925 03:01:32.958559 5663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:32.947098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45806 (local address 127.18.196.254:45949)
0925 03:01:32.948097 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:32.948108 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:32.948147 (+ 39us) server_negotiation.cc:407] Connection header received
0925 03:01:32.950181 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:32.950203 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:32.950622 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:32.951004 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:32.951578 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:32.952766 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:32.954218 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:32.955012 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:32.955206 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:32.956368 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:32.956375 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:32.956387 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:32.956886 (+ 499us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:32.957013 (+ 127us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:32.957035 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:32.957159 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:32.957371 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 03:01:32.957663 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":373,"threads_started":1}
I20260925 03:01:32.964475 5660 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0806e55a1f14e339555e76c52c1c656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0806e55a1f14e339555e76c52c1c656" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 45949 } }
I20260925 03:01:32.964831 5660 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0806e55a1f14e339555e76c52c1c656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:32.964943 5660 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0806e55a1f14e339555e76c52c1c656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0806e55a1f14e339555e76c52c1c656, State: Initialized, Role: FOLLOWER
I20260925 03:01:32.965466 5660 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0806e55a1f14e339555e76c52c1c656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0806e55a1f14e339555e76c52c1c656" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 45949 } }
I20260925 03:01:32.965709 5660 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0806e55a1f14e339555e76c52c1c656 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:01:32.965850 5660 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0806e55a1f14e339555e76c52c1c656 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:01:32.966061 5660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0806e55a1f14e339555e76c52c1c656 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:32.968312 5660 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0806e55a1f14e339555e76c52c1c656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0806e55a1f14e339555e76c52c1c656" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 45949 } }
I20260925 03:01:32.968801 5660 leader_election.cc:302] T 00000000000000000000000000000000 P b0806e55a1f14e339555e76c52c1c656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0806e55a1f14e339555e76c52c1c656; no voters:
I20260925 03:01:32.969497 5660 leader_election.cc:288] T 00000000000000000000000000000000 P b0806e55a1f14e339555e76c52c1c656 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:01:32.969647 5665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0806e55a1f14e339555e76c52c1c656 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:01:32.970394 5665 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0806e55a1f14e339555e76c52c1c656 [term 1 LEADER]: Becoming Leader. State: Replica: b0806e55a1f14e339555e76c52c1c656, State: Running, Role: LEADER
I20260925 03:01:32.970991 5665 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0806e55a1f14e339555e76c52c1c656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0806e55a1f14e339555e76c52c1c656" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 45949 } }
I20260925 03:01:32.971709 5660 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0806e55a1f14e339555e76c52c1c656 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:01:32.974025 5667 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0806e55a1f14e339555e76c52c1c656 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0806e55a1f14e339555e76c52c1c656. Latest consensus state: current_term: 1 leader_uuid: "b0806e55a1f14e339555e76c52c1c656" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0806e55a1f14e339555e76c52c1c656" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 45949 } } }
I20260925 03:01:32.974507 5667 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0806e55a1f14e339555e76c52c1c656 [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:32.974408 5666 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0806e55a1f14e339555e76c52c1c656 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0806e55a1f14e339555e76c52c1c656" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0806e55a1f14e339555e76c52c1c656" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 45949 } } }
I20260925 03:01:32.974712 5666 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0806e55a1f14e339555e76c52c1c656 [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:32.976279 5674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:01:32.981462 5674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:01:32.989900 5674 catalog_manager.cc:1347] Generated new cluster ID: 31cdb905d2114645b41ea5a135c73b69
I20260925 03:01:32.990008 5674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:01:33.010764 5674 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:01:33.012046 5674 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:01:33.018026 5674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0806e55a1f14e339555e76c52c1c656: Generated new TSK 0
I20260925 03:01:33.018783 5674 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:01:33.027076 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:45949
--builtin_ntp_servers=127.18.196.212:42013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 03:01:33.403167 5684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:33.403427 5684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:33.403509 5684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:33.410226 5684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:33.410307 5684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:33.410398 5684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 03:01:33.419134 5684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:42013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:45949
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:33.420262 5684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:33.421891 5684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:33.434525 5689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:33.434480 5692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:33.434571 5690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:33.435819 5684 server_base.cc:1034] running on GCE node
I20260925 03:01:33.436398 5684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:33.437327 5684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:33.438549 5684 hybrid_clock.cc:648] HybridClock initialized: now 1790305293438479 us; error 51 us; skew 500 ppm
I20260925 03:01:33.438930 5684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:01:33.441756 5684 webserver.cc:460] Webserver started at http://127.18.196.193:36821/ using document root <none> and password file <none>
I20260925 03:01:33.442684 5684 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:33.442821 5684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:33.443228 5684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:33.447381 5684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "bfa7f3247c3343bba8d11498c88df31a"
format_stamp: "Formatted at 2026-09-25 03:01:33 on dist-test-slave-8zvk"
server_key: "58ecdc0ba1e20472c8512d0c96297d7e"
server_key_iv: "a486db35eefa704a9c7e5d37f626649a"
server_key_version: "encryptionkey@0"
I20260925 03:01:33.448459 5684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "bfa7f3247c3343bba8d11498c88df31a"
format_stamp: "Formatted at 2026-09-25 03:01:33 on dist-test-slave-8zvk"
server_key: "58ecdc0ba1e20472c8512d0c96297d7e"
server_key_iv: "a486db35eefa704a9c7e5d37f626649a"
server_key_version: "encryptionkey@0"
I20260925 03:01:33.453521 5684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 03:01:33.456737 5698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:33.457999 5684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 03:01:33.458123 5684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "bfa7f3247c3343bba8d11498c88df31a"
format_stamp: "Formatted at 2026-09-25 03:01:33 on dist-test-slave-8zvk"
server_key: "58ecdc0ba1e20472c8512d0c96297d7e"
server_key_iv: "a486db35eefa704a9c7e5d37f626649a"
server_key_version: "encryptionkey@0"
I20260925 03:01:33.458287 5684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:33.473448 5684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:33.474225 5684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:33.474503 5684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:33.475636 5684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:01:33.477005 5684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:01:33.477087 5684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:33.477226 5684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:01:33.477306 5684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:33.510409 5684 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:42789
I20260925 03:01:33.510481 5810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:42789 every 8 connection(s)
I20260925 03:01:33.511765 5684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 03:01:33.516816 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 5684
I20260925 03:01:33.517023 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 03:01:33.517592 19219 external_mini_cluster.cc:1428] Setting key 72c6f6218bc82e58e27b0726bc035754
I20260925 03:01:33.520505 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:45949
--builtin_ntp_servers=127.18.196.212:42013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 03:01:33.525584 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:33.514227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:47595 (local address 127.18.196.254:45949)
0925 03:01:33.514719 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:33.514724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:33.515813 (+ 1089us) server_negotiation.cc:407] Connection header received
0925 03:01:33.518292 (+ 2479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:33.518297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:33.518390 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:33.518513 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:33.519608 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:33.520305 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:33.522813 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:33.523266 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:33.523362 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:33.524850 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:33.524855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:33.524858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:33.524875 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:33.524919 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:33.524931 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:33.524999 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:33.525261 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 03:01:33.525335 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":259,"threads_started":1}
I20260925 03:01:33.526995 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:33.514697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:45949 (local address 127.18.196.193:47595)
0925 03:01:33.515651 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:33.515706 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:33.517933 (+ 2227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:33.518740 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:33.518747 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:33.518824 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:01:33.519414 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:33.519431 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:33.520462 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:33.520467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:33.522675 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:33.522693 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:33.523491 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:33.523500 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:33.523807 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:33.524513 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 03:01:33.524518 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:01:33.524570 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 03:01:33.524664 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:01:33.525028 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:01:33.525042 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:01:33.525053 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 03:01:33.525246 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 03:01:33.525640 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":327,"threads_started":1}
I20260925 03:01:33.528414 5811 heartbeater.cc:347] Connected to a master server at 127.18.196.254:45949
I20260925 03:01:33.528714 5811 heartbeater.cc:464] Registering TS with master...
I20260925 03:01:33.529424 5811 heartbeater.cc:511] Master 127.18.196.254:45949 requested a full tablet report, sending...
I20260925 03:01:33.531510 5624 ts_manager.cc:194] Registered new tserver with Master: bfa7f3247c3343bba8d11498c88df31a (127.18.196.193:42789)
I20260925 03:01:33.532935 5624 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.196.193:47595
W20260925 03:01:33.843524 5815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:33.843773 5815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:33.843938 5815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:33.851023 5815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:33.851127 5815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:33.851298 5815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 03:01:33.859797 5815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:42013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:45949
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:33.860962 5815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:33.862731 5815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:33.875417 5820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:33.875348 5821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:33.875346 5823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:33.876148 5815 server_base.cc:1034] running on GCE node
I20260925 03:01:33.876700 5815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:33.877636 5815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:33.878923 5815 hybrid_clock.cc:648] HybridClock initialized: now 1790305293878867 us; error 65 us; skew 500 ppm
I20260925 03:01:33.879228 5815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:01:33.881428 5815 webserver.cc:460] Webserver started at http://127.18.196.194:46551/ using document root <none> and password file <none>
I20260925 03:01:33.882084 5815 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:33.882206 5815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:33.882503 5815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:33.885229 5815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "71dbd4c4f5c74f88a4a8a6e94ddc3223"
format_stamp: "Formatted at 2026-09-25 03:01:33 on dist-test-slave-8zvk"
server_key: "89f0c7c45281fb429cce9c5e69d7e512"
server_key_iv: "83b8f979290b2dd9280d4d60096d12a5"
server_key_version: "encryptionkey@0"
I20260925 03:01:33.885996 5815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "71dbd4c4f5c74f88a4a8a6e94ddc3223"
format_stamp: "Formatted at 2026-09-25 03:01:33 on dist-test-slave-8zvk"
server_key: "89f0c7c45281fb429cce9c5e69d7e512"
server_key_iv: "83b8f979290b2dd9280d4d60096d12a5"
server_key_version: "encryptionkey@0"
I20260925 03:01:33.890708 5815 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260925 03:01:33.894073 5829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:33.895361 5815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 03:01:33.895534 5815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "71dbd4c4f5c74f88a4a8a6e94ddc3223"
format_stamp: "Formatted at 2026-09-25 03:01:33 on dist-test-slave-8zvk"
server_key: "89f0c7c45281fb429cce9c5e69d7e512"
server_key_iv: "83b8f979290b2dd9280d4d60096d12a5"
server_key_version: "encryptionkey@0"
I20260925 03:01:33.895743 5815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:33.913019 5815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:33.913769 5815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:33.914073 5815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:33.915035 5815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:01:33.916425 5815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:01:33.916546 5815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:33.916678 5815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:01:33.916779 5815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:33.949162 5815 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:32955
I20260925 03:01:33.949208 5942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:32955 every 8 connection(s)
I20260925 03:01:33.950367 5815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 03:01:33.957872 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 5815
I20260925 03:01:33.958036 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 03:01:33.958491 19219 external_mini_cluster.cc:1428] Setting key a3daedee78abd168b6e4b67443fdcf38
I20260925 03:01:33.961006 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:45949
--builtin_ntp_servers=127.18.196.212:42013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 03:01:33.962613 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:33.952795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:38299 (local address 127.18.196.254:45949)
0925 03:01:33.952981 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:33.952987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:33.954183 (+ 1196us) server_negotiation.cc:407] Connection header received
0925 03:01:33.956169 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:33.956178 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:33.956266 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:33.956359 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:33.957315 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:33.958097 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:33.959775 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:33.960290 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:33.960412 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:33.961635 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:33.961644 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:33.961651 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:33.961676 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:33.961717 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:33.961736 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:33.961858 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:33.962090 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 03:01:33.962198 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 03:01:33.963449 5945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:33.953204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:45949 (local address 127.18.196.194:38299)
0925 03:01:33.954038 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:33.954071 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:33.955903 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:33.956564 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:33.956571 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:33.956642 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:01:33.957186 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:33.957201 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:33.958254 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:33.958258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:33.959627 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:33.959638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:33.960489 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:33.960496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:33.960699 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:33.961318 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 03:01:33.961321 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:01:33.961359 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 03:01:33.961431 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:01:33.961895 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:01:33.961903 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:01:33.961909 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 03:01:33.962045 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 03:01:33.962358 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":262,"threads_started":1}
I20260925 03:01:33.964702 5943 heartbeater.cc:347] Connected to a master server at 127.18.196.254:45949
I20260925 03:01:33.964980 5943 heartbeater.cc:464] Registering TS with master...
I20260925 03:01:33.965668 5943 heartbeater.cc:511] Master 127.18.196.254:45949 requested a full tablet report, sending...
I20260925 03:01:33.966969 5624 ts_manager.cc:194] Registered new tserver with Master: 71dbd4c4f5c74f88a4a8a6e94ddc3223 (127.18.196.194:32955)
I20260925 03:01:33.967744 5624 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.196.194:38299
W20260925 03:01:34.276150 5946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:34.276386 5946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:34.276468 5946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:34.283517 5946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:34.283601 5946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:34.283697 5946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 03:01:34.292279 5946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:42013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:45949
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:34.293414 5946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:34.295063 5946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:34.307510 5951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:34.307488 5952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:34.308211 5954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:34.308925 5946 server_base.cc:1034] running on GCE node
I20260925 03:01:34.309412 5946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:34.310221 5946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:34.311401 5946 hybrid_clock.cc:648] HybridClock initialized: now 1790305294311347 us; error 51 us; skew 500 ppm
I20260925 03:01:34.311681 5946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:01:34.313675 5946 webserver.cc:460] Webserver started at http://127.18.196.195:46197/ using document root <none> and password file <none>
I20260925 03:01:34.314286 5946 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:34.314399 5946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:34.314689 5946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:34.317467 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "9f68f2326271427488e05294e11e5c42"
format_stamp: "Formatted at 2026-09-25 03:01:34 on dist-test-slave-8zvk"
server_key: "079e03894ad0d8085f0b272727dc61fb"
server_key_iv: "4d72a90b76d13f4b4fb75d17c35208d0"
server_key_version: "encryptionkey@0"
I20260925 03:01:34.318216 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "9f68f2326271427488e05294e11e5c42"
format_stamp: "Formatted at 2026-09-25 03:01:34 on dist-test-slave-8zvk"
server_key: "079e03894ad0d8085f0b272727dc61fb"
server_key_iv: "4d72a90b76d13f4b4fb75d17c35208d0"
server_key_version: "encryptionkey@0"
I20260925 03:01:34.322885 5946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 03:01:34.325979 5960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:34.327234 5946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260925 03:01:34.327394 5946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "9f68f2326271427488e05294e11e5c42"
format_stamp: "Formatted at 2026-09-25 03:01:34 on dist-test-slave-8zvk"
server_key: "079e03894ad0d8085f0b272727dc61fb"
server_key_iv: "4d72a90b76d13f4b4fb75d17c35208d0"
server_key_version: "encryptionkey@0"
I20260925 03:01:34.327589 5946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:34.341400 5946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:34.342109 5946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:34.342394 5946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:34.343281 5946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:01:34.344525 5946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:01:34.344636 5946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:34.344753 5946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:01:34.344831 5946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:34.375793 5946 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:36789
I20260925 03:01:34.375877 6073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:36789 every 8 connection(s)
I20260925 03:01:34.376976 5946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 03:01:34.386231 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 5946
I20260925 03:01:34.386427 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 03:01:34.386927 19219 external_mini_cluster.cc:1428] Setting key 2db429a360faf22275210d0d0df64bd1
I20260925 03:01:34.388507 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:34.379990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:47679 (local address 127.18.196.254:45949)
0925 03:01:34.380192 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:34.380199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:34.380612 (+ 413us) server_negotiation.cc:407] Connection header received
0925 03:01:34.382316 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:34.382322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:34.382413 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:34.382519 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:34.383557 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.384288 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.385763 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.386186 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.386315 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:34.387716 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:34.387723 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:34.387728 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:34.387772 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:34.387815 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:34.387835 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:34.387929 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:34.388175 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 03:01:34.388263 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 03:01:34.389628 6076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:34.379631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:45949 (local address 127.18.196.195:47679)
0925 03:01:34.380485 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:34.380517 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:34.382071 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:34.382730 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:34.382738 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:34.382814 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:01:34.383403 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:34.383420 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.384427 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.384432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:34.385610 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:34.385622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.386401 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.386409 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:34.386662 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:34.387377 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 03:01:34.387382 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:01:34.387431 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 03:01:34.387516 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:01:34.387948 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:01:34.387959 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:01:34.387967 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 03:01:34.388149 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 03:01:34.388491 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":329,"threads_started":1}
I20260925 03:01:34.390726 6074 heartbeater.cc:347] Connected to a master server at 127.18.196.254:45949
I20260925 03:01:34.391044 6074 heartbeater.cc:464] Registering TS with master...
I20260925 03:01:34.391739 6074 heartbeater.cc:511] Master 127.18.196.254:45949 requested a full tablet report, sending...
I20260925 03:01:34.393064 5624 ts_manager.cc:194] Registered new tserver with Master: 9f68f2326271427488e05294e11e5c42 (127.18.196.195:36789)
I20260925 03:01:34.393946 5624 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.196.195:47679
I20260925 03:01:34.401824 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 03:01:34.411532 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:34.405936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45812 (local address 127.18.196.254:45949)
0925 03:01:34.406196 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:34.406202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:34.406474 (+ 272us) server_negotiation.cc:407] Connection header received
0925 03:01:34.406597 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:34.406601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:34.406668 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:34.406747 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:34.407231 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.407929 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.409253 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.409617 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.409702 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:34.410735 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:34.410739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:34.410742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:34.410757 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:34.410800 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:34.410811 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:34.410991 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:34.411180 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 03:01:34.411252 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 03:01:34.415139 5624 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 03:01:34.421887 5624 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 03:01:34.458029 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:34.433382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:36789 (local address 127.0.0.1:33396)
0925 03:01:34.450223 (+ 16841us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:34.450244 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:34.450389 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:34.451387 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:34.451394 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:34.451423 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:34.452107 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:34.452123 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.453905 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.453909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:34.455804 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:34.455819 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.457168 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.457175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:34.457365 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:34.457411 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 03:01:34.457539 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 03:01:34.457692 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16685,"mutex_wait_us":204,"thread_start_us":17548,"threads_started":1}
I20260925 03:01:34.458086 6089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:34.433862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33396 (local address 127.18.196.195:36789)
0925 03:01:34.449200 (+ 15338us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:34.449212 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:34.450355 (+ 1143us) server_negotiation.cc:407] Connection header received
0925 03:01:34.450574 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:34.450586 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:34.450903 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:34.451093 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:34.452305 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.453740 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.455995 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.456991 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.457169 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:34.457361 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:34.457595 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 03:01:34.457741 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15119,"thread_start_us":15349,"threads_started":1}
I20260925 03:01:34.459290 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:34.433185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:32955 (local address 127.0.0.1:45744)
0925 03:01:34.448349 (+ 15164us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:34.448399 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:34.448586 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:34.451446 (+ 2860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:34.451451 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:34.451486 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:34.452187 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:34.452205 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.453767 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.453772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:34.456595 (+ 2823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:34.456604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.458529 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.458533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:34.458665 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:34.458725 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 03:01:34.458814 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 03:01:34.458965 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14897,"mutex_wait_us":277,"thread_start_us":15591,"threads_started":1}
I20260925 03:01:34.459491 6087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:34.434161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:42789 (local address 127.0.0.1:41332)
0925 03:01:34.451135 (+ 16974us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:34.451151 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:34.451257 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:34.451807 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:34.451812 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:34.451839 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:34.452382 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:34.452391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.454027 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.454033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:34.456584 (+ 2551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:34.456599 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.458507 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.458519 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:34.458913 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:34.458946 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 03:01:34.459036 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 03:01:34.459150 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16878,"mutex_wait_us":404,"thread_start_us":15739,"threads_started":1}
I20260925 03:01:34.460088 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:34.434816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41332 (local address 127.18.196.193:42789)
0925 03:01:34.449746 (+ 14930us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:34.449757 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:34.451261 (+ 1504us) server_negotiation.cc:407] Connection header received
0925 03:01:34.451415 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:34.451421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:34.451613 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:34.451804 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:34.452542 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.453892 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.456807 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.458031 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.458187 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:34.458364 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:34.459612 (+ 1248us) server_negotiation.cc:299] Negotiation successful
0925 03:01:34.459765 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14789,"thread_start_us":14662,"threads_started":1}
I20260925 03:01:34.460589 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:34.433530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45744 (local address 127.18.196.194:32955)
0925 03:01:34.450904 (+ 17374us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:34.450912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:34.450941 (+ 29us) server_negotiation.cc:407] Connection header received
0925 03:01:34.451045 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:34.451052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:34.451269 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:34.451446 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:34.452398 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.453629 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.456800 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.458039 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.458221 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:34.458412 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:34.460076 (+ 1664us) server_negotiation.cc:299] Negotiation successful
0925 03:01:34.460217 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17187,"thread_start_us":17065,"threads_started":1}
I20260925 03:01:34.462342 6009 tablet_service.cc:1467] Processing CreateTablet for tablet a37d928dda2d41c0bc7566dfa63fcc85 (DEFAULT_TABLE table=test-table [id=c6a96812556d4e93ad329c07ebeab281]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 03:01:34.463595 6009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37d928dda2d41c0bc7566dfa63fcc85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:34.463398 5746 tablet_service.cc:1467] Processing CreateTablet for tablet a37d928dda2d41c0bc7566dfa63fcc85 (DEFAULT_TABLE table=test-table [id=c6a96812556d4e93ad329c07ebeab281]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 03:01:34.463963 5877 tablet_service.cc:1467] Processing CreateTablet for tablet a37d928dda2d41c0bc7566dfa63fcc85 (DEFAULT_TABLE table=test-table [id=c6a96812556d4e93ad329c07ebeab281]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 03:01:34.465107 5746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37d928dda2d41c0bc7566dfa63fcc85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:34.465142 5877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37d928dda2d41c0bc7566dfa63fcc85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:34.475378 6092 tablet_bootstrap.cc:492] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42: Bootstrap starting.
I20260925 03:01:34.478143 6092 tablet_bootstrap.cc:654] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:34.478904 6094 tablet_bootstrap.cc:492] T a37d928dda2d41c0bc7566dfa63fcc85 P 71dbd4c4f5c74f88a4a8a6e94ddc3223: Bootstrap starting.
I20260925 03:01:34.479545 6092 log.cc:824] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:34.482148 6094 tablet_bootstrap.cc:654] T a37d928dda2d41c0bc7566dfa63fcc85 P 71dbd4c4f5c74f88a4a8a6e94ddc3223: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:34.483050 6092 tablet_bootstrap.cc:492] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42: No bootstrap required, opened a new log
I20260925 03:01:34.483345 6092 ts_tablet_manager.cc:1397] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 03:01:34.483332 6094 log.cc:824] T a37d928dda2d41c0bc7566dfa63fcc85 P 71dbd4c4f5c74f88a4a8a6e94ddc3223: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:34.484710 6093 tablet_bootstrap.cc:492] T a37d928dda2d41c0bc7566dfa63fcc85 P bfa7f3247c3343bba8d11498c88df31a: Bootstrap starting.
I20260925 03:01:34.486672 6094 tablet_bootstrap.cc:492] T a37d928dda2d41c0bc7566dfa63fcc85 P 71dbd4c4f5c74f88a4a8a6e94ddc3223: No bootstrap required, opened a new log
I20260925 03:01:34.486966 6094 ts_tablet_manager.cc:1397] T a37d928dda2d41c0bc7566dfa63fcc85 P 71dbd4c4f5c74f88a4a8a6e94ddc3223: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260925 03:01:34.488029 6093 tablet_bootstrap.cc:654] T a37d928dda2d41c0bc7566dfa63fcc85 P bfa7f3247c3343bba8d11498c88df31a: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:34.489084 6093 log.cc:824] T a37d928dda2d41c0bc7566dfa63fcc85 P bfa7f3247c3343bba8d11498c88df31a: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:34.491832 6093 tablet_bootstrap.cc:492] T a37d928dda2d41c0bc7566dfa63fcc85 P bfa7f3247c3343bba8d11498c88df31a: No bootstrap required, opened a new log
I20260925 03:01:34.492090 6093 ts_tablet_manager.cc:1397] T a37d928dda2d41c0bc7566dfa63fcc85 P bfa7f3247c3343bba8d11498c88df31a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 03:01:34.493351 6092 raft_consensus.cc:348] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71dbd4c4f5c74f88a4a8a6e94ddc3223" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 32955 } } peers { permanent_uuid: "9f68f2326271427488e05294e11e5c42" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 36789 } } peers { permanent_uuid: "bfa7f3247c3343bba8d11498c88df31a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42789 } }
I20260925 03:01:34.493779 6092 raft_consensus.cc:374] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:34.493880 6092 raft_consensus.cc:729] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f68f2326271427488e05294e11e5c42, State: Initialized, Role: FOLLOWER
I20260925 03:01:34.493911 6094 raft_consensus.cc:348] T a37d928dda2d41c0bc7566dfa63fcc85 P 71dbd4c4f5c74f88a4a8a6e94ddc3223 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71dbd4c4f5c74f88a4a8a6e94ddc3223" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 32955 } } peers { permanent_uuid: "9f68f2326271427488e05294e11e5c42" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 36789 } } peers { permanent_uuid: "bfa7f3247c3343bba8d11498c88df31a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42789 } }
I20260925 03:01:34.494360 6094 raft_consensus.cc:374] T a37d928dda2d41c0bc7566dfa63fcc85 P 71dbd4c4f5c74f88a4a8a6e94ddc3223 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:34.494475 6094 raft_consensus.cc:729] T a37d928dda2d41c0bc7566dfa63fcc85 P 71dbd4c4f5c74f88a4a8a6e94ddc3223 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71dbd4c4f5c74f88a4a8a6e94ddc3223, State: Initialized, Role: FOLLOWER
I20260925 03:01:34.494473 6092 consensus_queue.cc:260] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71dbd4c4f5c74f88a4a8a6e94ddc3223" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 32955 } } peers { permanent_uuid: "9f68f2326271427488e05294e11e5c42" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 36789 } } peers { permanent_uuid: "bfa7f3247c3343bba8d11498c88df31a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42789 } }
I20260925 03:01:34.495055 6094 consensus_queue.cc:260] T a37d928dda2d41c0bc7566dfa63fcc85 P 71dbd4c4f5c74f88a4a8a6e94ddc3223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71dbd4c4f5c74f88a4a8a6e94ddc3223" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 32955 } } peers { permanent_uuid: "9f68f2326271427488e05294e11e5c42" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 36789 } } peers { permanent_uuid: "bfa7f3247c3343bba8d11498c88df31a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42789 } }
I20260925 03:01:34.495682 6074 heartbeater.cc:503] Master 127.18.196.254:45949 was elected leader, sending a full tablet report...
I20260925 03:01:34.496271 6092 ts_tablet_manager.cc:1428] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
I20260925 03:01:34.497982 5943 heartbeater.cc:503] Master 127.18.196.254:45949 was elected leader, sending a full tablet report...
I20260925 03:01:34.500453 6094 ts_tablet_manager.cc:1428] T a37d928dda2d41c0bc7566dfa63fcc85 P 71dbd4c4f5c74f88a4a8a6e94ddc3223: Time spent starting tablet: real 0.013s user 0.010s sys 0.001s
I20260925 03:01:34.501880 6093 raft_consensus.cc:348] T a37d928dda2d41c0bc7566dfa63fcc85 P bfa7f3247c3343bba8d11498c88df31a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71dbd4c4f5c74f88a4a8a6e94ddc3223" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 32955 } } peers { permanent_uuid: "9f68f2326271427488e05294e11e5c42" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 36789 } } peers { permanent_uuid: "bfa7f3247c3343bba8d11498c88df31a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42789 } }
I20260925 03:01:34.502211 6093 raft_consensus.cc:374] T a37d928dda2d41c0bc7566dfa63fcc85 P bfa7f3247c3343bba8d11498c88df31a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:34.502336 6093 raft_consensus.cc:729] T a37d928dda2d41c0bc7566dfa63fcc85 P bfa7f3247c3343bba8d11498c88df31a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfa7f3247c3343bba8d11498c88df31a, State: Initialized, Role: FOLLOWER
I20260925 03:01:34.502983 6093 consensus_queue.cc:260] T a37d928dda2d41c0bc7566dfa63fcc85 P bfa7f3247c3343bba8d11498c88df31a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71dbd4c4f5c74f88a4a8a6e94ddc3223" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 32955 } } peers { permanent_uuid: "9f68f2326271427488e05294e11e5c42" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 36789 } } peers { permanent_uuid: "bfa7f3247c3343bba8d11498c88df31a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42789 } }
I20260925 03:01:34.503932 5811 heartbeater.cc:503] Master 127.18.196.254:45949 was elected leader, sending a full tablet report...
I20260925 03:01:34.504487 6093 ts_tablet_manager.cc:1428] T a37d928dda2d41c0bc7566dfa63fcc85 P bfa7f3247c3343bba8d11498c88df31a: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
W20260925 03:01:34.513962 5812 tablet.cc:2369] T a37d928dda2d41c0bc7566dfa63fcc85 P bfa7f3247c3343bba8d11498c88df31a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 03:01:34.528782 6098 raft_consensus.cc:482] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:01:34.529079 6098 raft_consensus.cc:504] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71dbd4c4f5c74f88a4a8a6e94ddc3223" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 32955 } } peers { permanent_uuid: "9f68f2326271427488e05294e11e5c42" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 36789 } } peers { permanent_uuid: "bfa7f3247c3343bba8d11498c88df31a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42789 } }
I20260925 03:01:34.530669 6098 leader_election.cc:288] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71dbd4c4f5c74f88a4a8a6e94ddc3223 (127.18.196.194:32955), bfa7f3247c3343bba8d11498c88df31a (127.18.196.193:42789)
I20260925 03:01:34.536892 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:34.530974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:54729 (local address 127.18.196.194:32955)
0925 03:01:34.531191 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:34.531197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:34.531360 (+ 163us) server_negotiation.cc:407] Connection header received
0925 03:01:34.531546 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:34.531550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:34.531607 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:34.531709 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:34.532194 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.532941 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.534940 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.536121 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.536280 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:34.536336 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:34.536639 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 03:01:34.536722 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 03:01:34.537086 6076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:34.531073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:32955 (local address 127.18.196.195:54729)
0925 03:01:34.531258 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:34.531279 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:34.531405 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:34.531849 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:34.531854 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:34.531889 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:34.532050 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:34.532058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.533078 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.533084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:34.534770 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:34.534784 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.536257 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.536265 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:34.536487 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:34.536524 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 03:01:34.536618 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 03:01:34.536751 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":5}
I20260925 03:01:34.538139 6101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:34.531378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:42789 (local address 127.18.196.195:46467)
0925 03:01:34.532045 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:34.532057 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:34.532183 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:34.532489 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:34.532493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:34.532515 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:34.532799 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:34.532807 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.534074 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.534077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:34.536032 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:34.536049 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.537583 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.537592 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:34.537744 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:34.537767 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 03:01:34.537853 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 03:01:34.537929 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":374,"threads_started":1}
I20260925 03:01:34.538300 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:34.531208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:46467 (local address 127.18.196.193:42789)
0925 03:01:34.531400 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:34.531406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:34.532137 (+ 731us) server_negotiation.cc:407] Connection header received
0925 03:01:34.532305 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:34.532310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:34.532374 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:34.532458 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:34.532932 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.533929 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.536235 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.537438 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.537546 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:34.537587 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:34.537905 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 03:01:34.538049 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 03:01:34.538134 5897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37d928dda2d41c0bc7566dfa63fcc85" candidate_uuid: "9f68f2326271427488e05294e11e5c42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71dbd4c4f5c74f88a4a8a6e94ddc3223" is_pre_election: true
I20260925 03:01:34.538656 5897 raft_consensus.cc:2454] T a37d928dda2d41c0bc7566dfa63fcc85 P 71dbd4c4f5c74f88a4a8a6e94ddc3223 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f68f2326271427488e05294e11e5c42 in term 0.
I20260925 03:01:34.539232 5766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37d928dda2d41c0bc7566dfa63fcc85" candidate_uuid: "9f68f2326271427488e05294e11e5c42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfa7f3247c3343bba8d11498c88df31a" is_pre_election: true
I20260925 03:01:34.539368 5963 leader_election.cc:302] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71dbd4c4f5c74f88a4a8a6e94ddc3223, 9f68f2326271427488e05294e11e5c42; no voters:
I20260925 03:01:34.539644 5766 raft_consensus.cc:2454] T a37d928dda2d41c0bc7566dfa63fcc85 P bfa7f3247c3343bba8d11498c88df31a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f68f2326271427488e05294e11e5c42 in term 0.
I20260925 03:01:34.539732 6098 raft_consensus.cc:2781] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:01:34.539888 6098 raft_consensus.cc:482] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:01:34.540009 6098 raft_consensus.cc:3037] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:34.542136 6098 raft_consensus.cc:504] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71dbd4c4f5c74f88a4a8a6e94ddc3223" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 32955 } } peers { permanent_uuid: "9f68f2326271427488e05294e11e5c42" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 36789 } } peers { permanent_uuid: "bfa7f3247c3343bba8d11498c88df31a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42789 } }
I20260925 03:01:34.542735 6098 leader_election.cc:288] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42 [CANDIDATE]: Term 1 election: Requested vote from peers 71dbd4c4f5c74f88a4a8a6e94ddc3223 (127.18.196.194:32955), bfa7f3247c3343bba8d11498c88df31a (127.18.196.193:42789)
I20260925 03:01:34.543073 5897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37d928dda2d41c0bc7566dfa63fcc85" candidate_uuid: "9f68f2326271427488e05294e11e5c42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71dbd4c4f5c74f88a4a8a6e94ddc3223"
I20260925 03:01:34.543253 5897 raft_consensus.cc:3037] T a37d928dda2d41c0bc7566dfa63fcc85 P 71dbd4c4f5c74f88a4a8a6e94ddc3223 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:34.543241 5766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37d928dda2d41c0bc7566dfa63fcc85" candidate_uuid: "9f68f2326271427488e05294e11e5c42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfa7f3247c3343bba8d11498c88df31a"
I20260925 03:01:34.543417 5766 raft_consensus.cc:3037] T a37d928dda2d41c0bc7566dfa63fcc85 P bfa7f3247c3343bba8d11498c88df31a [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:34.545378 5897 raft_consensus.cc:2454] T a37d928dda2d41c0bc7566dfa63fcc85 P 71dbd4c4f5c74f88a4a8a6e94ddc3223 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f68f2326271427488e05294e11e5c42 in term 1.
I20260925 03:01:34.545542 5766 raft_consensus.cc:2454] T a37d928dda2d41c0bc7566dfa63fcc85 P bfa7f3247c3343bba8d11498c88df31a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f68f2326271427488e05294e11e5c42 in term 1.
I20260925 03:01:34.545751 5963 leader_election.cc:302] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71dbd4c4f5c74f88a4a8a6e94ddc3223, 9f68f2326271427488e05294e11e5c42; no voters:
I20260925 03:01:34.546028 6098 raft_consensus.cc:2781] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:01:34.546465 6098 raft_consensus.cc:686] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42 [term 1 LEADER]: Becoming Leader. State: Replica: 9f68f2326271427488e05294e11e5c42, State: Running, Role: LEADER
I20260925 03:01:34.546950 6098 consensus_queue.cc:237] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71dbd4c4f5c74f88a4a8a6e94ddc3223" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 32955 } } peers { permanent_uuid: "9f68f2326271427488e05294e11e5c42" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 36789 } } peers { permanent_uuid: "bfa7f3247c3343bba8d11498c88df31a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42789 } }
I20260925 03:01:34.551301 5622 catalog_manager.cc:5495] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f68f2326271427488e05294e11e5c42 (127.18.196.195). New cstate: current_term: 1 leader_uuid: "9f68f2326271427488e05294e11e5c42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71dbd4c4f5c74f88a4a8a6e94ddc3223" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f68f2326271427488e05294e11e5c42" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 36789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bfa7f3247c3343bba8d11498c88df31a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42789 } health_report { overall_health: UNKNOWN } } }
I20260925 03:01:34.570379 6089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:34.565603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33404 (local address 127.18.196.195:36789)
0925 03:01:34.565785 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:34.565789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:34.565806 (+ 17us) server_negotiation.cc:407] Connection header received
0925 03:01:34.565936 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:34.565940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:34.566011 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:34.566123 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:01:34.566475 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.567213 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.568508 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.568869 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.568950 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:34.569380 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:01:34.569807 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:01:34.569946 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:34.570062 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 03:01:34.570119 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 03:01:34.577811 5897 raft_consensus.cc:1261] T a37d928dda2d41c0bc7566dfa63fcc85 P 71dbd4c4f5c74f88a4a8a6e94ddc3223 [term 1 FOLLOWER]: Refusing update from remote peer 9f68f2326271427488e05294e11e5c42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:01:34.577811 5766 raft_consensus.cc:1261] T a37d928dda2d41c0bc7566dfa63fcc85 P bfa7f3247c3343bba8d11498c88df31a [term 1 FOLLOWER]: Refusing update from remote peer 9f68f2326271427488e05294e11e5c42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:01:34.578635 6102 consensus_queue.cc:1035] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfa7f3247c3343bba8d11498c88df31a" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:01:34.578984 6098 consensus_queue.cc:1035] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71dbd4c4f5c74f88a4a8a6e94ddc3223" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:01:34.589111 6105 mvcc.cc:205] Tried to move back new op lower bound from 7333090486581669888 to 7333090486467637248. Current Snapshot: MvccSnapshot[applied={T|T < 7333090486581669888}]
I20260925 03:01:34.607304 5623 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 03:01:34.607673 5623 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 03:01:34.611274 5623 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0806e55a1f14e339555e76c52c1c656: Sending DeleteTablet for 3 replicas of tablet a37d928dda2d41c0bc7566dfa63fcc85
I20260925 03:01:34.612370 6009 tablet_service.cc:1514] Processing DeleteTablet for tablet a37d928dda2d41c0bc7566dfa63fcc85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 03:01:34 UTC) from {username='slave'} at 127.0.0.1:33396
I20260925 03:01:34.612376 5877 tablet_service.cc:1514] Processing DeleteTablet for tablet a37d928dda2d41c0bc7566dfa63fcc85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 03:01:34 UTC) from {username='slave'} at 127.0.0.1:45744
I20260925 03:01:34.613524 5746 tablet_service.cc:1514] Processing DeleteTablet for tablet a37d928dda2d41c0bc7566dfa63fcc85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 03:01:34 UTC) from {username='slave'} at 127.0.0.1:41332
I20260925 03:01:34.616354 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 5684
I20260925 03:01:34.627792 6116 tablet_replica.cc:330] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42: stopping tablet replica
I20260925 03:01:34.628793 6116 raft_consensus.cc:2229] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42 [term 1 LEADER]: Raft consensus shutting down.
I20260925 03:01:34.629730 6116 raft_consensus.cc:2258] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:01:34.632407 6116 ts_tablet_manager.cc:1905] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 03:01:34.641409 6116 ts_tablet_manager.cc:1918] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 03:01:34.641655 6116 log.cc:1196] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/a37d928dda2d41c0bc7566dfa63fcc85
I20260925 03:01:34.642375 6116 ts_tablet_manager.cc:1939] T a37d928dda2d41c0bc7566dfa63fcc85 P 9f68f2326271427488e05294e11e5c42: Deleting consensus metadata
I20260925 03:01:34.642464 6118 tablet_replica.cc:330] T a37d928dda2d41c0bc7566dfa63fcc85 P bfa7f3247c3343bba8d11498c88df31a: stopping tablet replica
I20260925 03:01:34.644292 5611 catalog_manager.cc:4841] TS 9f68f2326271427488e05294e11e5c42 (127.18.196.195:36789): tablet a37d928dda2d41c0bc7566dfa63fcc85 (table test-table [id=c6a96812556d4e93ad329c07ebeab281]) successfully deleted
I20260925 03:01:34.645819 6118 raft_consensus.cc:2229] T a37d928dda2d41c0bc7566dfa63fcc85 P bfa7f3247c3343bba8d11498c88df31a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:01:34.646435 6118 raft_consensus.cc:2258] T a37d928dda2d41c0bc7566dfa63fcc85 P bfa7f3247c3343bba8d11498c88df31a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:01:34.647476 6117 tablet_replica.cc:330] T a37d928dda2d41c0bc7566dfa63fcc85 P 71dbd4c4f5c74f88a4a8a6e94ddc3223: stopping tablet replica
I20260925 03:01:34.647925 6117 raft_consensus.cc:2229] T a37d928dda2d41c0bc7566dfa63fcc85 P 71dbd4c4f5c74f88a4a8a6e94ddc3223 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:01:34.648437 6117 raft_consensus.cc:2258] T a37d928dda2d41c0bc7566dfa63fcc85 P 71dbd4c4f5c74f88a4a8a6e94ddc3223 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:01:34.649092 6118 ts_tablet_manager.cc:1905] T a37d928dda2d41c0bc7566dfa63fcc85 P bfa7f3247c3343bba8d11498c88df31a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 03:01:34.649508 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:34.617287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41340 (local address 127.18.196.193:42789)
0925 03:01:34.617459 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:34.617465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:34.640625 (+ 23160us) server_negotiation.cc:407] Connection header received
0925 03:01:34.641089 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:34.641098 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:34.641223 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:34.641306 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:34.642647 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.643823 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.645416 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.646095 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.646211 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:34.647755 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:34.647766 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:34.647783 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:34.648542 (+ 759us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:34.648607 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:34.648634 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:34.648718 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:34.649000 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 03:01:34.649118 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 03:01:34.650657 6117 ts_tablet_manager.cc:1905] T a37d928dda2d41c0bc7566dfa63fcc85 P 71dbd4c4f5c74f88a4a8a6e94ddc3223: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 03:01:34.650898 5806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:34.657271 6117 ts_tablet_manager.cc:1918] T a37d928dda2d41c0bc7566dfa63fcc85 P 71dbd4c4f5c74f88a4a8a6e94ddc3223: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 03:01:34.657444 6117 log.cc:1196] T a37d928dda2d41c0bc7566dfa63fcc85 P 71dbd4c4f5c74f88a4a8a6e94ddc3223: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/a37d928dda2d41c0bc7566dfa63fcc85
I20260925 03:01:34.657913 6117 ts_tablet_manager.cc:1939] T a37d928dda2d41c0bc7566dfa63fcc85 P 71dbd4c4f5c74f88a4a8a6e94ddc3223: Deleting consensus metadata
I20260925 03:01:34.659361 5610 catalog_manager.cc:4841] TS 71dbd4c4f5c74f88a4a8a6e94ddc3223 (127.18.196.194:32955): tablet a37d928dda2d41c0bc7566dfa63fcc85 (table test-table [id=c6a96812556d4e93ad329c07ebeab281]) successfully deleted
I20260925 03:01:34.783744 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 5684
I20260925 03:01:34.802325 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 5815
W20260925 03:01:34.802299 5608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.196.193:42789 (error 108)
I20260925 03:01:34.804209 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:34.803810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:42789 (local address 127.0.0.1:41344)
0925 03:01:34.803981 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:34.804075 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 03:01:34.804533 5608 catalog_manager.cc:4584] TS bfa7f3247c3343bba8d11498c88df31a (127.18.196.193:42789): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a37d928dda2d41c0bc7566dfa63fcc85: Network error: Client connection negotiation failed: client connection to 127.18.196.193:42789: connect: Connection refused (error 111)
I20260925 03:01:34.810225 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:34.803907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45752 (local address 127.18.196.194:32955)
0925 03:01:34.804474 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:34.804478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:34.804494 (+ 16us) server_negotiation.cc:407] Connection header received
0925 03:01:34.804550 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:34.804554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:34.804644 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:34.804760 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:34.805179 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.805909 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.807348 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.807737 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.807868 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:34.808907 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:34.808916 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:34.808930 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:34.809497 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:34.809544 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:34.809564 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:34.809681 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:34.809810 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 03:01:34.809914 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460}
I20260925 03:01:34.810897 5938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:34.858292 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:34.857724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:42789 (local address 127.0.0.1:41354)
0925 03:01:34.857979 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:34.858098 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 03:01:34.859508 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:34.859055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:42789 (local address 127.0.0.1:41368)
0925 03:01:34.859224 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:34.859321 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 03:01:34.899595 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 5815
I20260925 03:01:34.920410 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:34.919974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:42789 (local address 127.0.0.1:41378)
0925 03:01:34.920166 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:34.920249 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 03:01:34.921547 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:34.921187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:42789 (local address 127.0.0.1:41394)
0925 03:01:34.921340 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:34.921413 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 03:01:34.923517 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 5946
I20260925 03:01:34.931578 6089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:34.924266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33418 (local address 127.18.196.195:36789)
0925 03:01:34.924479 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:34.924487 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:34.924514 (+ 27us) server_negotiation.cc:407] Connection header received
0925 03:01:34.924587 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:34.924593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:34.924708 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:34.924835 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:34.925203 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.926381 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.927753 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:34.928500 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:34.928647 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:34.929662 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:34.929674 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:34.929695 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:34.930590 (+ 895us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:34.930658 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:34.930692 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:34.930823 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:34.931097 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 03:01:34.931193 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 03:01:34.932262 6069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:35.002236 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:35.001777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:42789 (local address 127.0.0.1:41410)
0925 03:01:35.001996 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:35.002087 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 03:01:35.003360 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:35.002945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:42789 (local address 127.0.0.1:41422)
0925 03:01:35.003147 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:35.003251 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 03:01:35.061820 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 5946
I20260925 03:01:35.081962 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 5593
I20260925 03:01:35.083168 5655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:35.192847 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 5593
I20260925 03:01:35.192922 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:35.192223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:42789 (local address 127.0.0.1:41438)
0925 03:01:35.192614 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:35.192734 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
2026-09-25T03:01:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2745 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 03:01:35 dist-test-slave-8zvk krb5kdc[6127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 03:01:35 dist-test-slave-8zvk krb5kdc[6127](info): set up 2 sockets
Sep 25 03:01:35 dist-test-slave-8zvk krb5kdc[6127](info): commencing operation
krb5kdc: starting...
W20260925 03:01:36.298424 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 03:01:36 dist-test-slave-8zvk krb5kdc[6127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T03:01:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:01:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 03:01:36.554440 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:43129
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:35771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:43129
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:01:36.893566 6140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:36.893884 6140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:36.893977 6140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:36.901489 6140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:01:36.901602 6140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:36.901648 6140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:36.901707 6140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:01:36.901763 6140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:01:36.911584 6140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:35771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:43129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:43129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:36.912851 6140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:36.914069 6140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:36.925791 6146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:36.925916 6145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:36.925882 6148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:36.926878 6140 server_base.cc:1034] running on GCE node
I20260925 03:01:36.927610 6140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:36.929029 6140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:36.930233 6140 hybrid_clock.cc:648] HybridClock initialized: now 1790305296930196 us; error 29 us; skew 500 ppm
I20260925 03:01:36.930538 6140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:36 dist-test-slave-8zvk krb5kdc[6127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305296, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:36.938918 6140 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 03:01:36.940544 6140 webserver.cc:460] Webserver started at http://127.18.196.254:33287/ using document root <none> and password file <none>
I20260925 03:01:36.941215 6140 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:36.941322 6140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:36.941581 6140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:36.944664 6140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "0d3e4726a4eb49cab907291968f9cc4b"
format_stamp: "Formatted at 2026-09-25 03:01:36 on dist-test-slave-8zvk"
server_key: "4d36c3eea68e00a2de6471fee727f23d"
server_key_iv: "37cd930f83dd9406ede6622105efa464"
server_key_version: "encryptionkey@0"
I20260925 03:01:36.945395 6140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "0d3e4726a4eb49cab907291968f9cc4b"
format_stamp: "Formatted at 2026-09-25 03:01:36 on dist-test-slave-8zvk"
server_key: "4d36c3eea68e00a2de6471fee727f23d"
server_key_iv: "37cd930f83dd9406ede6622105efa464"
server_key_version: "encryptionkey@0"
I20260925 03:01:36.950521 6140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 03:01:36.953975 6155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:36.955358 6140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 03:01:36.955518 6140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "0d3e4726a4eb49cab907291968f9cc4b"
format_stamp: "Formatted at 2026-09-25 03:01:36 on dist-test-slave-8zvk"
server_key: "4d36c3eea68e00a2de6471fee727f23d"
server_key_iv: "37cd930f83dd9406ede6622105efa464"
server_key_version: "encryptionkey@0"
I20260925 03:01:36.955696 6140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:36.969094 6140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:36.973273 6140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:36.973618 6140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:36.992715 6140 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:43129
I20260925 03:01:36.992739 6206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:43129 every 8 connection(s)
I20260925 03:01:36.994226 6140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:01:36.998180 6207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:36.999853 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 6140
I20260925 03:01:37.000324 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:01:37.000819 19219 external_mini_cluster.cc:1428] Setting key 671ce9c48ca42a88f44e5bd4cd0dd817
I20260925 03:01:37.013111 6207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3e4726a4eb49cab907291968f9cc4b: Bootstrap starting.
Sep 25 03:01:37 dist-test-slave-8zvk krb5kdc[6127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:01:37.017050 6207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d3e4726a4eb49cab907291968f9cc4b: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:37.018244 6207 log.cc:824] T 00000000000000000000000000000000 P 0d3e4726a4eb49cab907291968f9cc4b: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:37.021302 6207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3e4726a4eb49cab907291968f9cc4b: No bootstrap required, opened a new log
I20260925 03:01:37.025074 6210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:37.002691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42788 (local address 127.18.196.254:43129)
0925 03:01:37.003583 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:37.003595 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:37.003637 (+ 42us) server_negotiation.cc:407] Connection header received
0925 03:01:37.005259 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:37.005282 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:37.005700 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:37.006017 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:37.008117 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:37.009536 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:37.011497 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:37.012168 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:37.012306 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:37.017249 (+ 4943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:37.017258 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:37.017273 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:37.017312 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:37.020729 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:37.021478 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:37.021485 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:37.021491 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:37.021658 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:37.022103 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:37.022108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:37.022110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:37.022637 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:37.023048 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:37.023599 (+ 551us) server_negotiation.cc:299] Negotiation successful
0925 03:01:37.024031 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":462,"threads_started":1}
I20260925 03:01:37.029605 6207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d3e4726a4eb49cab907291968f9cc4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3e4726a4eb49cab907291968f9cc4b" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43129 } }
I20260925 03:01:37.030081 6207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d3e4726a4eb49cab907291968f9cc4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:37.030189 6207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d3e4726a4eb49cab907291968f9cc4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3e4726a4eb49cab907291968f9cc4b, State: Initialized, Role: FOLLOWER
I20260925 03:01:37.030812 6207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d3e4726a4eb49cab907291968f9cc4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3e4726a4eb49cab907291968f9cc4b" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43129 } }
I20260925 03:01:37.031177 6207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d3e4726a4eb49cab907291968f9cc4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:01:37.031317 6207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d3e4726a4eb49cab907291968f9cc4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:01:37.031509 6207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d3e4726a4eb49cab907291968f9cc4b [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:37.034003 6207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d3e4726a4eb49cab907291968f9cc4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3e4726a4eb49cab907291968f9cc4b" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43129 } }
I20260925 03:01:37.034524 6207 leader_election.cc:302] T 00000000000000000000000000000000 P 0d3e4726a4eb49cab907291968f9cc4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d3e4726a4eb49cab907291968f9cc4b; no voters:
I20260925 03:01:37.035318 6207 leader_election.cc:288] T 00000000000000000000000000000000 P 0d3e4726a4eb49cab907291968f9cc4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:01:37.035429 6212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d3e4726a4eb49cab907291968f9cc4b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:01:37.036350 6212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d3e4726a4eb49cab907291968f9cc4b [term 1 LEADER]: Becoming Leader. State: Replica: 0d3e4726a4eb49cab907291968f9cc4b, State: Running, Role: LEADER
I20260925 03:01:37.036904 6212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d3e4726a4eb49cab907291968f9cc4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3e4726a4eb49cab907291968f9cc4b" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43129 } }
I20260925 03:01:37.037586 6207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d3e4726a4eb49cab907291968f9cc4b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:01:37.039891 6213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3e4726a4eb49cab907291968f9cc4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d3e4726a4eb49cab907291968f9cc4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3e4726a4eb49cab907291968f9cc4b" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43129 } } }
I20260925 03:01:37.040364 6213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3e4726a4eb49cab907291968f9cc4b [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:37.040544 6214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3e4726a4eb49cab907291968f9cc4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d3e4726a4eb49cab907291968f9cc4b. Latest consensus state: current_term: 1 leader_uuid: "0d3e4726a4eb49cab907291968f9cc4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3e4726a4eb49cab907291968f9cc4b" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43129 } } }
I20260925 03:01:37.040844 6214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3e4726a4eb49cab907291968f9cc4b [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:37.042876 6221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:01:37.047374 6221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:01:37.056702 6221 catalog_manager.cc:1347] Generated new cluster ID: fef2591c44bc46baae143a1fa8f0e4aa
I20260925 03:01:37.056839 6221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:01:37.085927 6221 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:01:37.088124 6221 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:01:37.098300 6221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d3e4726a4eb49cab907291968f9cc4b: Generated new TSK 0
I20260925 03:01:37.099363 6221 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
Entry for principal kudu/127.18.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab.
I20260925 03:01:37.172104 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:43129
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:35771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:01:37.517648 6233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:37.517966 6233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:37.518136 6233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:37.525707 6233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:37.525857 6233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:37.526047 6233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 03:01:37.536177 6233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:35771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.193.keytab
--principal=kudu/127.18.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:43129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:37.537545 6233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:37.538988 6233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:37.552217 6238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:37.552187 6241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:37.552256 6239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:37.553470 6233 server_base.cc:1034] running on GCE node
I20260925 03:01:37.554132 6233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:37.556341 6233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:37.557586 6233 hybrid_clock.cc:648] HybridClock initialized: now 1790305297557530 us; error 68 us; skew 500 ppm
I20260925 03:01:37.557946 6233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:37 dist-test-slave-8zvk krb5kdc[6127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305297, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:37.561286 6233 init.cc:377] Logged in from keytab as kudu/127.18.196.193@KRBTEST.COM (short username kudu)
I20260925 03:01:37.562805 6233 webserver.cc:460] Webserver started at http://127.18.196.193:39017/ using document root <none> and password file <none>
I20260925 03:01:37.563572 6233 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:37.563717 6233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:37.564043 6233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:37.567060 6233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "19974138dea64c4586021b1acbab0e81"
format_stamp: "Formatted at 2026-09-25 03:01:37 on dist-test-slave-8zvk"
server_key: "1405c18edc21123690f8e95f5853aa40"
server_key_iv: "fb4efa01039217dba211e2d4ddf72682"
server_key_version: "encryptionkey@0"
I20260925 03:01:37.567847 6233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "19974138dea64c4586021b1acbab0e81"
format_stamp: "Formatted at 2026-09-25 03:01:37 on dist-test-slave-8zvk"
server_key: "1405c18edc21123690f8e95f5853aa40"
server_key_iv: "fb4efa01039217dba211e2d4ddf72682"
server_key_version: "encryptionkey@0"
I20260925 03:01:37.573078 6233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 03:01:37.576606 6248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:37.577961 6233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 03:01:37.578145 6233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "19974138dea64c4586021b1acbab0e81"
format_stamp: "Formatted at 2026-09-25 03:01:37 on dist-test-slave-8zvk"
server_key: "1405c18edc21123690f8e95f5853aa40"
server_key_iv: "fb4efa01039217dba211e2d4ddf72682"
server_key_version: "encryptionkey@0"
I20260925 03:01:37.578357 6233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:37.596967 6233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:37.601374 6233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:37.601728 6233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:37.602726 6233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:01:37.604175 6233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:01:37.604250 6233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:37.604409 6233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:01:37.604467 6233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:37.637844 6233 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:39175
I20260925 03:01:37.637882 6361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:39175 every 8 connection(s)
I20260925 03:01:37.639358 6233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 03:01:37.643062 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 6233
I20260925 03:01:37.643276 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 03:01:37.643863 19219 external_mini_cluster.cc:1428] Setting key 3e2feba4f60b381cbad2c3757279806a
Sep 25 03:01:37 dist-test-slave-8zvk krb5kdc[6127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305297, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.193@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:01:37.663618 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:37.642016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:35823 (local address 127.18.196.254:43129)
0925 03:01:37.642529 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:37.642537 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:37.643701 (+ 1164us) server_negotiation.cc:407] Connection header received
0925 03:01:37.645987 (+ 2286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:37.645991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:37.646056 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:37.646149 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:37.648275 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:37.648992 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:37.651438 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:37.651849 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:37.651957 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:37.656298 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:37.656306 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:37.656312 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:37.656363 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:37.660055 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:37.660842 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:37.660851 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:37.660858 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:37.661023 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:37.661480 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:37.661488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:37.661494 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:37.661904 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:37.662073 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:37.662800 (+ 727us) server_negotiation.cc:299] Negotiation successful
0925 03:01:37.663115 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":235,"threads_started":1}
I20260925 03:01:37.664316 6365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:37.642609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:43129 (local address 127.18.196.193:35823)
0925 03:01:37.643505 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:37.643558 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:37.645673 (+ 2115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:37.646430 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:37.646444 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:37.647254 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:01:37.648091 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:37.648120 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:37.649149 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:37.649158 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:37.651280 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:37.651299 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:37.652103 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:37.652112 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:37.652454 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:37.653216 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:01:37.653249 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:01:37.656015 (+ 2766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:01:37.660213 (+ 4198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:01:37.660221 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:01:37.660236 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:01:37.660695 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:01:37.661146 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:01:37.661150 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:01:37.661152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:01:37.661356 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:01:37.662038 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:01:37.662045 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:01:37.662402 (+ 357us) client_negotiation.cc:769] Sending connection context
0925 03:01:37.662635 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 03:01:37.663133 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":335,"threads_started":1}
I20260925 03:01:37.666015 6362 heartbeater.cc:347] Connected to a master server at 127.18.196.254:43129
I20260925 03:01:37.666347 6362 heartbeater.cc:464] Registering TS with master...
I20260925 03:01:37.667270 6362 heartbeater.cc:511] Master 127.18.196.254:43129 requested a full tablet report, sending...
I20260925 03:01:37.669785 6172 ts_manager.cc:194] Registered new tserver with Master: 19974138dea64c4586021b1acbab0e81 (127.18.196.193:39175)
I20260925 03:01:37.672113 6172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.193@KRBTEST.COM'} at 127.18.196.193:35823
WARNING: no policy specified for kudu/127.18.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
Entry for principal kudu/127.18.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab.
I20260925 03:01:37.709566 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:43129
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:35771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:01:38.029564 6368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:38.029807 6368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:38.029894 6368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:38.036736 6368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:38.036830 6368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:38.036952 6368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 03:01:38.047145 6368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:35771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.194.keytab
--principal=kudu/127.18.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:43129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:38.048287 6368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:38.049538 6368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:38.062361 6373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:38.062403 6374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:38.062458 6376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:38.063222 6368 server_base.cc:1034] running on GCE node
I20260925 03:01:38.063812 6368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:38.064666 6368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:38.065868 6368 hybrid_clock.cc:648] HybridClock initialized: now 1790305298065791 us; error 67 us; skew 500 ppm
I20260925 03:01:38.066156 6368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:38 dist-test-slave-8zvk krb5kdc[6127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305298, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:38.068984 6368 init.cc:377] Logged in from keytab as kudu/127.18.196.194@KRBTEST.COM (short username kudu)
I20260925 03:01:38.070279 6368 webserver.cc:460] Webserver started at http://127.18.196.194:43173/ using document root <none> and password file <none>
I20260925 03:01:38.070973 6368 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:38.071110 6368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:38.071401 6368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:38.074069 6368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "0fc806b7cbba42068c514a5753da7609"
format_stamp: "Formatted at 2026-09-25 03:01:38 on dist-test-slave-8zvk"
server_key: "12896f51508e1c9bb18ac9cb4bc7ac30"
server_key_iv: "e7905c2a6fd330fd2d6ba733d9e5677b"
server_key_version: "encryptionkey@0"
I20260925 03:01:38.074807 6368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "0fc806b7cbba42068c514a5753da7609"
format_stamp: "Formatted at 2026-09-25 03:01:38 on dist-test-slave-8zvk"
server_key: "12896f51508e1c9bb18ac9cb4bc7ac30"
server_key_iv: "e7905c2a6fd330fd2d6ba733d9e5677b"
server_key_version: "encryptionkey@0"
I20260925 03:01:38.079991 6368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 03:01:38.083665 6383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:38.084918 6368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 03:01:38.085084 6368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "0fc806b7cbba42068c514a5753da7609"
format_stamp: "Formatted at 2026-09-25 03:01:38 on dist-test-slave-8zvk"
server_key: "12896f51508e1c9bb18ac9cb4bc7ac30"
server_key_iv: "e7905c2a6fd330fd2d6ba733d9e5677b"
server_key_version: "encryptionkey@0"
I20260925 03:01:38.085278 6368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:38.112267 6368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:38.115968 6368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:38.116252 6368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:38.117179 6368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:01:38.118510 6368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:01:38.118606 6368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:38.118736 6368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:01:38.118821 6368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:38.151244 6368 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:43941
I20260925 03:01:38.151322 6495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:43941 every 8 connection(s)
I20260925 03:01:38.152516 6368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 03:01:38.155151 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 6368
I20260925 03:01:38.155354 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 03:01:38.155818 19219 external_mini_cluster.cc:1428] Setting key 38a3457b7aa436b19ba0e3e161ed861a
Sep 25 03:01:38 dist-test-slave-8zvk krb5kdc[6127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305298, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.194@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:01:38.175338 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:38.155054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:49555 (local address 127.18.196.254:43129)
0925 03:01:38.155270 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:38.155276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:38.156482 (+ 1206us) server_negotiation.cc:407] Connection header received
0925 03:01:38.158823 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:38.158828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:38.158933 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:38.159013 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:38.161016 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.161700 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.164105 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.164508 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.164598 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:38.168881 (+ 4283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:38.168886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:38.168889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:38.168926 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:38.172266 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:38.172965 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:38.172973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:38.172978 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:38.173138 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:38.173525 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:38.173532 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:38.173538 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:38.173886 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:38.174025 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:38.174640 (+ 615us) server_negotiation.cc:299] Negotiation successful
0925 03:01:38.174910 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 03:01:38.176035 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:38.155515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:43129 (local address 127.18.196.194:49555)
0925 03:01:38.156304 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:38.156355 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:38.158502 (+ 2147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:38.159270 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:38.159284 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:38.160037 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:01:38.160844 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:38.160871 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.161868 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.161877 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:38.163970 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:38.163990 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.164720 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.164729 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:38.165070 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:38.165762 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:01:38.165794 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:01:38.168639 (+ 2845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:01:38.172392 (+ 3753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:01:38.172398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:01:38.172409 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:01:38.172795 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:01:38.173250 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:01:38.173253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:01:38.173255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:01:38.173417 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:01:38.174002 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:01:38.174008 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:01:38.174279 (+ 271us) client_negotiation.cc:769] Sending connection context
0925 03:01:38.174480 (+ 201us) client_negotiation.cc:240] Negotiation successful
0925 03:01:38.174872 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":339,"threads_started":1}
I20260925 03:01:38.177549 6496 heartbeater.cc:347] Connected to a master server at 127.18.196.254:43129
I20260925 03:01:38.177873 6496 heartbeater.cc:464] Registering TS with master...
I20260925 03:01:38.178680 6496 heartbeater.cc:511] Master 127.18.196.254:43129 requested a full tablet report, sending...
I20260925 03:01:38.180259 6172 ts_manager.cc:194] Registered new tserver with Master: 0fc806b7cbba42068c514a5753da7609 (127.18.196.194:43941)
I20260925 03:01:38.181473 6172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.194@KRBTEST.COM'} at 127.18.196.194:49555
WARNING: no policy specified for kudu/127.18.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
Entry for principal kudu/127.18.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab.
I20260925 03:01:38.216126 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:43129
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:35771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:01:38.540807 6501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:38.541095 6501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:38.541296 6501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:38.548601 6501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:38.548727 6501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:38.548885 6501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 03:01:38.558283 6501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:35771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.195.keytab
--principal=kudu/127.18.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:43129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:38.559583 6501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:38.560868 6501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:38.573544 6506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:38.573567 6509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:38.573526 6507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:38.574298 6501 server_base.cc:1034] running on GCE node
I20260925 03:01:38.574959 6501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:38.575879 6501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:38.577091 6501 hybrid_clock.cc:648] HybridClock initialized: now 1790305298577044 us; error 52 us; skew 500 ppm
I20260925 03:01:38.577381 6501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:38 dist-test-slave-8zvk krb5kdc[6127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305298, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:38.580480 6501 init.cc:377] Logged in from keytab as kudu/127.18.196.195@KRBTEST.COM (short username kudu)
I20260925 03:01:38.581916 6501 webserver.cc:460] Webserver started at http://127.18.196.195:46013/ using document root <none> and password file <none>
I20260925 03:01:38.582580 6501 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:38.582713 6501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:38.583077 6501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:38.585848 6501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "f14baf89a61e474a9952fabbf3eca69f"
format_stamp: "Formatted at 2026-09-25 03:01:38 on dist-test-slave-8zvk"
server_key: "d5c1cc0228b74bbb7ffd853ddab5840c"
server_key_iv: "958b2b9e505f9b1bb1cda08bad253843"
server_key_version: "encryptionkey@0"
I20260925 03:01:38.586601 6501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "f14baf89a61e474a9952fabbf3eca69f"
format_stamp: "Formatted at 2026-09-25 03:01:38 on dist-test-slave-8zvk"
server_key: "d5c1cc0228b74bbb7ffd853ddab5840c"
server_key_iv: "958b2b9e505f9b1bb1cda08bad253843"
server_key_version: "encryptionkey@0"
I20260925 03:01:38.591750 6501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 03:01:38.595402 6516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:38.596750 6501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 03:01:38.596942 6501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "f14baf89a61e474a9952fabbf3eca69f"
format_stamp: "Formatted at 2026-09-25 03:01:38 on dist-test-slave-8zvk"
server_key: "d5c1cc0228b74bbb7ffd853ddab5840c"
server_key_iv: "958b2b9e505f9b1bb1cda08bad253843"
server_key_version: "encryptionkey@0"
I20260925 03:01:38.597147 6501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:38.609552 6501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:38.613698 6501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:38.613994 6501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:38.614993 6501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:01:38.616277 6501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:01:38.616377 6501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:38.616511 6501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:01:38.616580 6501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:38.651988 6501 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:35687
I20260925 03:01:38.652027 6628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:35687 every 8 connection(s)
I20260925 03:01:38.653245 6501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 03:01:38.659936 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 6501
I20260925 03:01:38.660192 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 03:01:38.660588 19219 external_mini_cluster.cc:1428] Setting key ffebe628029d619155d7af17f09fae26
Sep 25 03:01:38 dist-test-slave-8zvk krb5kdc[6127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305298, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.195@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:01:38.672843 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:38.655838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:42503 (local address 127.18.196.254:43129)
0925 03:01:38.656046 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:38.656050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:38.657166 (+ 1116us) server_negotiation.cc:407] Connection header received
0925 03:01:38.659051 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:38.659057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:38.659173 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:38.659285 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:38.660873 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.661955 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.663767 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.664162 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.664289 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:38.667653 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:38.667658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:38.667661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:38.667695 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:38.670129 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:38.670812 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:38.670817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:38.670819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:38.671013 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:38.671397 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:38.671400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:38.671402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:38.671662 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:38.671770 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:38.672440 (+ 670us) server_negotiation.cc:299] Negotiation successful
0925 03:01:38.672598 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 03:01:38.673913 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:38.656296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:43129 (local address 127.18.196.195:42503)
0925 03:01:38.657002 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:38.657039 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:38.658725 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:38.659521 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:38.659549 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:38.660122 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:01:38.660695 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:38.660713 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.662106 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.662111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:38.663615 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:38.663632 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.664406 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.664414 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:38.664640 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:38.665259 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:01:38.665281 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:01:38.667460 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:01:38.670258 (+ 2798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:01:38.670265 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:01:38.670286 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:01:38.670706 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:01:38.671108 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:01:38.671111 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:01:38.671114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:01:38.671300 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:01:38.671777 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:01:38.671784 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:01:38.672101 (+ 317us) client_negotiation.cc:769] Sending connection context
0925 03:01:38.672335 (+ 234us) client_negotiation.cc:240] Negotiation successful
0925 03:01:38.672713 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":268,"threads_started":1}
I20260925 03:01:38.675232 6629 heartbeater.cc:347] Connected to a master server at 127.18.196.254:43129
I20260925 03:01:38.675568 6629 heartbeater.cc:464] Registering TS with master...
I20260925 03:01:38.676314 6362 heartbeater.cc:503] Master 127.18.196.254:43129 was elected leader, sending a full tablet report...
I20260925 03:01:38.676373 6629 heartbeater.cc:511] Master 127.18.196.254:43129 requested a full tablet report, sending...
I20260925 03:01:38.677764 6171 ts_manager.cc:194] Registered new tserver with Master: f14baf89a61e474a9952fabbf3eca69f (127.18.196.195:35687)
I20260925 03:01:38.678586 6171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.196.195@KRBTEST.COM'} at 127.18.196.195:42503
I20260925 03:01:38.687841 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 03:01:38.704627 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:38.692389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42802 (local address 127.18.196.254:43129)
0925 03:01:38.692622 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:38.692629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:38.692960 (+ 331us) server_negotiation.cc:407] Connection header received
0925 03:01:38.693085 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:38.693089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:38.693154 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:38.693229 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:38.694578 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.695485 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.696775 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.697179 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.697269 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:38.699698 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:38.699702 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:38.699705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:38.699737 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:38.702232 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:38.702830 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:38.702880 (+ 50us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:38.702883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:38.703020 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:38.703330 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:38.703333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:38.703335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:38.703653 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:38.703834 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:38.704109 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 03:01:38.704264 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 03:01:38.708570 6171 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 03:01:38.715272 6171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 03:01:38.752691 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:38.727800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:35687 (local address 127.0.0.1:46272)
0925 03:01:38.742804 (+ 15004us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:38.742826 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:38.742991 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:38.745912 (+ 2921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:38.745917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:38.745954 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:38.746380 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:38.746396 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.748076 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.748083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:38.749802 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:38.749814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.751673 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.751679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:38.751861 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:38.751927 (+ 66us) client_negotiation.cc:769] Sending connection context
0925 03:01:38.752043 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 03:01:38.752221 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14851,"mutex_wait_us":796,"thread_start_us":431,"threads_started":1}
I20260925 03:01:38.753213 6645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:38.728092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46272 (local address 127.18.196.195:35687)
0925 03:01:38.745344 (+ 17252us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:38.745355 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:38.745388 (+ 33us) server_negotiation.cc:407] Connection header received
0925 03:01:38.745482 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:38.745489 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:38.745708 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:38.746475 (+ 767us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:38.746736 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.747943 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.749960 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.751442 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.752463 (+ 1021us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:38.752623 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:38.752691 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 03:01:38.752855 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17075,"thread_start_us":16904,"threads_started":1}
I20260925 03:01:38.753233 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:38.727347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:43941 (local address 127.0.0.1:58762)
0925 03:01:38.729246 (+ 1899us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:38.729268 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:38.729405 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:38.746445 (+ 17040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:38.746449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:38.746476 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:38.746792 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:38.746799 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.748668 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.748674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:38.750534 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:38.750545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.752444 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.752451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:38.752631 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:38.752666 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 03:01:38.752766 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 03:01:38.752890 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1716,"thread_start_us":489,"threads_started":1}
I20260925 03:01:38.752846 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:38.726934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:39175 (local address 127.0.0.1:37084)
0925 03:01:38.728156 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:38.728205 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:38.728425 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:38.744564 (+ 16139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:38.744576 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:38.744620 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:38.745093 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:38.745110 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.747361 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.747367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:38.749952 (+ 2585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:38.749968 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.752019 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.752028 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:38.752193 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:38.752222 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 03:01:38.752301 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 03:01:38.752449 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"mutex_wait_us":98,"thread_start_us":16137,"threads_started":1}
I20260925 03:01:38.753489 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:38.727157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37084 (local address 127.18.196.193:39175)
0925 03:01:38.743747 (+ 16590us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:38.743761 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:38.743799 (+ 38us) server_negotiation.cc:407] Connection header received
0925 03:01:38.743939 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:38.743951 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:38.744282 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:38.744529 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:38.745351 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.747220 (+ 1869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.750108 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.751504 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.751688 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:38.751868 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:38.752958 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0925 03:01:38.753124 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16397,"thread_start_us":16252,"threads_started":1}
I20260925 03:01:38.753970 6644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:38.727409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58762 (local address 127.18.196.194:43941)
0925 03:01:38.745636 (+ 18227us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:38.745643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:38.745667 (+ 24us) server_negotiation.cc:407] Connection header received
0925 03:01:38.745759 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:38.745766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:38.745972 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:38.746164 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:38.746983 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.748447 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.750681 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.751945 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.752111 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:38.752272 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:38.753472 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0925 03:01:38.753635 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18073,"thread_start_us":17940,"threads_started":1}
I20260925 03:01:38.756901 6564 tablet_service.cc:1467] Processing CreateTablet for tablet da54372469a14afb88d33d05a27670d5 (DEFAULT_TABLE table=test-table [id=a45f6a75031846abaf61bf32a4f051dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 03:01:38.757176 6431 tablet_service.cc:1467] Processing CreateTablet for tablet da54372469a14afb88d33d05a27670d5 (DEFAULT_TABLE table=test-table [id=a45f6a75031846abaf61bf32a4f051dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 03:01:38.757171 6296 tablet_service.cc:1467] Processing CreateTablet for tablet da54372469a14afb88d33d05a27670d5 (DEFAULT_TABLE table=test-table [id=a45f6a75031846abaf61bf32a4f051dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 03:01:38.758203 6564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da54372469a14afb88d33d05a27670d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:38.758489 6431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da54372469a14afb88d33d05a27670d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:38.758529 6296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da54372469a14afb88d33d05a27670d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:38.769918 6647 tablet_bootstrap.cc:492] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f: Bootstrap starting.
I20260925 03:01:38.772892 6648 tablet_bootstrap.cc:492] T da54372469a14afb88d33d05a27670d5 P 19974138dea64c4586021b1acbab0e81: Bootstrap starting.
I20260925 03:01:38.773479 6647 tablet_bootstrap.cc:654] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:38.774574 6649 tablet_bootstrap.cc:492] T da54372469a14afb88d33d05a27670d5 P 0fc806b7cbba42068c514a5753da7609: Bootstrap starting.
I20260925 03:01:38.775089 6647 log.cc:824] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:38.776283 6648 tablet_bootstrap.cc:654] T da54372469a14afb88d33d05a27670d5 P 19974138dea64c4586021b1acbab0e81: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:38.777580 6649 tablet_bootstrap.cc:654] T da54372469a14afb88d33d05a27670d5 P 0fc806b7cbba42068c514a5753da7609: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:38.777526 6648 log.cc:824] T da54372469a14afb88d33d05a27670d5 P 19974138dea64c4586021b1acbab0e81: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:38.778406 6647 tablet_bootstrap.cc:492] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f: No bootstrap required, opened a new log
I20260925 03:01:38.778698 6649 log.cc:824] T da54372469a14afb88d33d05a27670d5 P 0fc806b7cbba42068c514a5753da7609: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:38.778961 6647 ts_tablet_manager.cc:1397] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260925 03:01:38.780508 6648 tablet_bootstrap.cc:492] T da54372469a14afb88d33d05a27670d5 P 19974138dea64c4586021b1acbab0e81: No bootstrap required, opened a new log
I20260925 03:01:38.780771 6648 ts_tablet_manager.cc:1397] T da54372469a14afb88d33d05a27670d5 P 19974138dea64c4586021b1acbab0e81: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 03:01:38.781749 6649 tablet_bootstrap.cc:492] T da54372469a14afb88d33d05a27670d5 P 0fc806b7cbba42068c514a5753da7609: No bootstrap required, opened a new log
I20260925 03:01:38.782021 6649 ts_tablet_manager.cc:1397] T da54372469a14afb88d33d05a27670d5 P 0fc806b7cbba42068c514a5753da7609: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260925 03:01:38.787053 6647 raft_consensus.cc:348] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19974138dea64c4586021b1acbab0e81" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 39175 } } peers { permanent_uuid: "0fc806b7cbba42068c514a5753da7609" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 43941 } } peers { permanent_uuid: "f14baf89a61e474a9952fabbf3eca69f" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 35687 } }
I20260925 03:01:38.787405 6647 raft_consensus.cc:374] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:38.787545 6647 raft_consensus.cc:729] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f14baf89a61e474a9952fabbf3eca69f, State: Initialized, Role: FOLLOWER
I20260925 03:01:38.788197 6647 consensus_queue.cc:260] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19974138dea64c4586021b1acbab0e81" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 39175 } } peers { permanent_uuid: "0fc806b7cbba42068c514a5753da7609" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 43941 } } peers { permanent_uuid: "f14baf89a61e474a9952fabbf3eca69f" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 35687 } }
I20260925 03:01:38.789273 6629 heartbeater.cc:503] Master 127.18.196.254:43129 was elected leader, sending a full tablet report...
I20260925 03:01:38.789988 6647 ts_tablet_manager.cc:1428] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260925 03:01:38.791340 6648 raft_consensus.cc:348] T da54372469a14afb88d33d05a27670d5 P 19974138dea64c4586021b1acbab0e81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19974138dea64c4586021b1acbab0e81" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 39175 } } peers { permanent_uuid: "0fc806b7cbba42068c514a5753da7609" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 43941 } } peers { permanent_uuid: "f14baf89a61e474a9952fabbf3eca69f" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 35687 } }
I20260925 03:01:38.792377 6648 raft_consensus.cc:374] T da54372469a14afb88d33d05a27670d5 P 19974138dea64c4586021b1acbab0e81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:38.792492 6648 raft_consensus.cc:729] T da54372469a14afb88d33d05a27670d5 P 19974138dea64c4586021b1acbab0e81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19974138dea64c4586021b1acbab0e81, State: Initialized, Role: FOLLOWER
I20260925 03:01:38.793119 6648 consensus_queue.cc:260] T da54372469a14afb88d33d05a27670d5 P 19974138dea64c4586021b1acbab0e81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19974138dea64c4586021b1acbab0e81" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 39175 } } peers { permanent_uuid: "0fc806b7cbba42068c514a5753da7609" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 43941 } } peers { permanent_uuid: "f14baf89a61e474a9952fabbf3eca69f" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 35687 } }
I20260925 03:01:38.793370 6649 raft_consensus.cc:348] T da54372469a14afb88d33d05a27670d5 P 0fc806b7cbba42068c514a5753da7609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19974138dea64c4586021b1acbab0e81" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 39175 } } peers { permanent_uuid: "0fc806b7cbba42068c514a5753da7609" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 43941 } } peers { permanent_uuid: "f14baf89a61e474a9952fabbf3eca69f" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 35687 } }
I20260925 03:01:38.793941 6649 raft_consensus.cc:374] T da54372469a14afb88d33d05a27670d5 P 0fc806b7cbba42068c514a5753da7609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:38.794078 6649 raft_consensus.cc:729] T da54372469a14afb88d33d05a27670d5 P 0fc806b7cbba42068c514a5753da7609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fc806b7cbba42068c514a5753da7609, State: Initialized, Role: FOLLOWER
I20260925 03:01:38.794884 6649 consensus_queue.cc:260] T da54372469a14afb88d33d05a27670d5 P 0fc806b7cbba42068c514a5753da7609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19974138dea64c4586021b1acbab0e81" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 39175 } } peers { permanent_uuid: "0fc806b7cbba42068c514a5753da7609" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 43941 } } peers { permanent_uuid: "f14baf89a61e474a9952fabbf3eca69f" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 35687 } }
I20260925 03:01:38.795684 6648 ts_tablet_manager.cc:1428] T da54372469a14afb88d33d05a27670d5 P 19974138dea64c4586021b1acbab0e81: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260925 03:01:38.796603 6496 heartbeater.cc:503] Master 127.18.196.254:43129 was elected leader, sending a full tablet report...
I20260925 03:01:38.797214 6649 ts_tablet_manager.cc:1428] T da54372469a14afb88d33d05a27670d5 P 0fc806b7cbba42068c514a5753da7609: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
I20260925 03:01:38.840680 6653 raft_consensus.cc:482] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:01:38.841001 6653 raft_consensus.cc:504] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19974138dea64c4586021b1acbab0e81" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 39175 } } peers { permanent_uuid: "0fc806b7cbba42068c514a5753da7609" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 43941 } } peers { permanent_uuid: "f14baf89a61e474a9952fabbf3eca69f" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 35687 } }
I20260925 03:01:38.842685 6653 leader_election.cc:288] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19974138dea64c4586021b1acbab0e81 (127.18.196.193:39175), 0fc806b7cbba42068c514a5753da7609 (127.18.196.194:43941)
I20260925 03:01:38.848490 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:38.843136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:52079 (local address 127.18.196.193:39175)
0925 03:01:38.843342 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:38.843348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:38.843473 (+ 125us) server_negotiation.cc:407] Connection header received
0925 03:01:38.843637 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:38.843640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:38.843700 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:38.843814 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:38.844222 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.845152 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.846800 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.847620 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.847780 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:38.847852 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:38.848129 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 03:01:38.848248 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 03:01:38.848500 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:38.843178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:39175 (local address 127.18.196.195:52079)
0925 03:01:38.843372 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:38.843390 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:38.843523 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:38.843832 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:38.843836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:38.843864 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:38.844077 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:38.844086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.845304 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.845309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:38.846650 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:38.846662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.847770 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.847776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:38.847965 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:38.847990 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 03:01:38.848089 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 03:01:38.848215 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":38}
I20260925 03:01:38.849414 6316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da54372469a14afb88d33d05a27670d5" candidate_uuid: "f14baf89a61e474a9952fabbf3eca69f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19974138dea64c4586021b1acbab0e81" is_pre_election: true
I20260925 03:01:38.849627 6644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:38.843213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:50299 (local address 127.18.196.194:43941)
0925 03:01:38.843427 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:38.843432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:38.844450 (+ 1018us) server_negotiation.cc:407] Connection header received
0925 03:01:38.844544 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:38.844549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:38.844630 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:38.844718 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:38.845336 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.846032 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.847767 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.848533 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.848715 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:38.848778 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:38.849255 (+ 477us) server_negotiation.cc:299] Negotiation successful
0925 03:01:38.849381 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 03:01:38.849717 6656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:38.843181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:43941 (local address 127.18.196.195:50299)
0925 03:01:38.844342 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:38.844355 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:38.844447 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:38.844774 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:38.844778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:38.844800 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:38.845140 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:38.845147 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.846190 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.846194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:38.847611 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:38.847624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.848904 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.848910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:38.849068 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:38.849099 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 03:01:38.849214 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 03:01:38.849347 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"mutex_wait_us":417,"thread_start_us":406,"threads_started":1}
I20260925 03:01:38.849944 6316 raft_consensus.cc:2454] T da54372469a14afb88d33d05a27670d5 P 19974138dea64c4586021b1acbab0e81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f14baf89a61e474a9952fabbf3eca69f in term 0.
I20260925 03:01:38.850664 6518 leader_election.cc:302] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19974138dea64c4586021b1acbab0e81, f14baf89a61e474a9952fabbf3eca69f; no voters:
I20260925 03:01:38.850661 6451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da54372469a14afb88d33d05a27670d5" candidate_uuid: "f14baf89a61e474a9952fabbf3eca69f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fc806b7cbba42068c514a5753da7609" is_pre_election: true
I20260925 03:01:38.851063 6653 raft_consensus.cc:2781] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:01:38.851186 6653 raft_consensus.cc:482] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:01:38.851276 6653 raft_consensus.cc:3037] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:38.851205 6451 raft_consensus.cc:2454] T da54372469a14afb88d33d05a27670d5 P 0fc806b7cbba42068c514a5753da7609 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f14baf89a61e474a9952fabbf3eca69f in term 0.
I20260925 03:01:38.853904 6653 raft_consensus.cc:504] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19974138dea64c4586021b1acbab0e81" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 39175 } } peers { permanent_uuid: "0fc806b7cbba42068c514a5753da7609" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 43941 } } peers { permanent_uuid: "f14baf89a61e474a9952fabbf3eca69f" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 35687 } }
I20260925 03:01:38.854450 6653 leader_election.cc:288] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f [CANDIDATE]: Term 1 election: Requested vote from peers 19974138dea64c4586021b1acbab0e81 (127.18.196.193:39175), 0fc806b7cbba42068c514a5753da7609 (127.18.196.194:43941)
I20260925 03:01:38.854960 6451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da54372469a14afb88d33d05a27670d5" candidate_uuid: "f14baf89a61e474a9952fabbf3eca69f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fc806b7cbba42068c514a5753da7609"
I20260925 03:01:38.855063 6316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da54372469a14afb88d33d05a27670d5" candidate_uuid: "f14baf89a61e474a9952fabbf3eca69f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19974138dea64c4586021b1acbab0e81"
I20260925 03:01:38.855146 6451 raft_consensus.cc:3037] T da54372469a14afb88d33d05a27670d5 P 0fc806b7cbba42068c514a5753da7609 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:38.855302 6316 raft_consensus.cc:3037] T da54372469a14afb88d33d05a27670d5 P 19974138dea64c4586021b1acbab0e81 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:38.857290 6451 raft_consensus.cc:2454] T da54372469a14afb88d33d05a27670d5 P 0fc806b7cbba42068c514a5753da7609 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f14baf89a61e474a9952fabbf3eca69f in term 1.
I20260925 03:01:38.857555 6316 raft_consensus.cc:2454] T da54372469a14afb88d33d05a27670d5 P 19974138dea64c4586021b1acbab0e81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f14baf89a61e474a9952fabbf3eca69f in term 1.
I20260925 03:01:38.857725 6519 leader_election.cc:302] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fc806b7cbba42068c514a5753da7609, f14baf89a61e474a9952fabbf3eca69f; no voters:
I20260925 03:01:38.858048 6653 raft_consensus.cc:2781] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:01:38.858487 6653 raft_consensus.cc:686] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f [term 1 LEADER]: Becoming Leader. State: Replica: f14baf89a61e474a9952fabbf3eca69f, State: Running, Role: LEADER
I20260925 03:01:38.859202 6653 consensus_queue.cc:237] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19974138dea64c4586021b1acbab0e81" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 39175 } } peers { permanent_uuid: "0fc806b7cbba42068c514a5753da7609" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 43941 } } peers { permanent_uuid: "f14baf89a61e474a9952fabbf3eca69f" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 35687 } }
I20260925 03:01:38.863828 6172 catalog_manager.cc:5495] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f reported cstate change: term changed from 0 to 1, leader changed from <none> to f14baf89a61e474a9952fabbf3eca69f (127.18.196.195). New cstate: current_term: 1 leader_uuid: "f14baf89a61e474a9952fabbf3eca69f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19974138dea64c4586021b1acbab0e81" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 39175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fc806b7cbba42068c514a5753da7609" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 43941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f14baf89a61e474a9952fabbf3eca69f" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 35687 } health_report { overall_health: HEALTHY } } }
W20260925 03:01:38.891877 6363 tablet.cc:2369] T da54372469a14afb88d33d05a27670d5 P 19974138dea64c4586021b1acbab0e81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 03:01:38.894447 6645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:38.888974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46276 (local address 127.18.196.195:35687)
0925 03:01:38.889136 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:38.889140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:38.889157 (+ 17us) server_negotiation.cc:407] Connection header received
0925 03:01:38.889299 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:38.889302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:38.889380 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:38.889560 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:01:38.889946 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.890675 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.892380 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.892773 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.892894 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:38.893339 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:01:38.893851 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:01:38.893962 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:38.894096 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 03:01:38.894165 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 03:01:38.902827 6316 raft_consensus.cc:1261] T da54372469a14afb88d33d05a27670d5 P 19974138dea64c4586021b1acbab0e81 [term 1 FOLLOWER]: Refusing update from remote peer f14baf89a61e474a9952fabbf3eca69f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:01:38.902849 6451 raft_consensus.cc:1261] T da54372469a14afb88d33d05a27670d5 P 0fc806b7cbba42068c514a5753da7609 [term 1 FOLLOWER]: Refusing update from remote peer f14baf89a61e474a9952fabbf3eca69f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:01:38.903801 6657 consensus_queue.cc:1035] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f [LEADER]: Connected to new peer: Peer: permanent_uuid: "19974138dea64c4586021b1acbab0e81" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 39175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:01:38.904140 6653 consensus_queue.cc:1035] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fc806b7cbba42068c514a5753da7609" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 43941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 03:01:38.904704 6497 tablet.cc:2369] T da54372469a14afb88d33d05a27670d5 P 0fc806b7cbba42068c514a5753da7609: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 03:01:38.904840 6630 tablet.cc:2369] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 03:01:38.916795 6662 mvcc.cc:205] Tried to move back new op lower bound from 7333090504295710720 to 7333090504132378624. Current Snapshot: MvccSnapshot[applied={T|T < 7333090504295710720}]
I20260925 03:01:38.928784 6171 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 03:01:38.929103 6171 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 03:01:38.932771 6171 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d3e4726a4eb49cab907291968f9cc4b: Sending DeleteTablet for 3 replicas of tablet da54372469a14afb88d33d05a27670d5
I20260925 03:01:38.933981 6296 tablet_service.cc:1514] Processing DeleteTablet for tablet da54372469a14afb88d33d05a27670d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 03:01:38 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:37084
I20260925 03:01:38.933966 6431 tablet_service.cc:1514] Processing DeleteTablet for tablet da54372469a14afb88d33d05a27670d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 03:01:38 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:58762
I20260925 03:01:38.934124 6564 tablet_service.cc:1514] Processing DeleteTablet for tablet da54372469a14afb88d33d05a27670d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 03:01:38 UTC) from {username='kudu', principal='kudu/127.18.196.254@KRBTEST.COM'} at 127.0.0.1:46272
I20260925 03:01:38.938603 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 6233
I20260925 03:01:38.946583 6671 tablet_replica.cc:330] T da54372469a14afb88d33d05a27670d5 P 0fc806b7cbba42068c514a5753da7609: stopping tablet replica
I20260925 03:01:38.947151 6671 raft_consensus.cc:2229] T da54372469a14afb88d33d05a27670d5 P 0fc806b7cbba42068c514a5753da7609 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:01:38.947768 6671 raft_consensus.cc:2258] T da54372469a14afb88d33d05a27670d5 P 0fc806b7cbba42068c514a5753da7609 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:01:38.950297 6671 ts_tablet_manager.cc:1905] T da54372469a14afb88d33d05a27670d5 P 0fc806b7cbba42068c514a5753da7609: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 03:01:38.958884 6671 ts_tablet_manager.cc:1918] T da54372469a14afb88d33d05a27670d5 P 0fc806b7cbba42068c514a5753da7609: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 03:01:38.959170 6671 log.cc:1196] T da54372469a14afb88d33d05a27670d5 P 0fc806b7cbba42068c514a5753da7609: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/da54372469a14afb88d33d05a27670d5
I20260925 03:01:38.959772 6671 ts_tablet_manager.cc:1939] T da54372469a14afb88d33d05a27670d5 P 0fc806b7cbba42068c514a5753da7609: Deleting consensus metadata
I20260925 03:01:38.961834 6158 catalog_manager.cc:4841] TS 0fc806b7cbba42068c514a5753da7609 (127.18.196.194:43941): tablet da54372469a14afb88d33d05a27670d5 (table test-table [id=a45f6a75031846abaf61bf32a4f051dc]) successfully deleted
I20260925 03:01:38.963155 6672 tablet_replica.cc:330] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f: stopping tablet replica
I20260925 03:01:38.963370 6670 tablet_replica.cc:330] T da54372469a14afb88d33d05a27670d5 P 19974138dea64c4586021b1acbab0e81: stopping tablet replica
I20260925 03:01:38.963828 6672 raft_consensus.cc:2229] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f [term 1 LEADER]: Raft consensus shutting down.
I20260925 03:01:38.963847 6670 raft_consensus.cc:2229] T da54372469a14afb88d33d05a27670d5 P 19974138dea64c4586021b1acbab0e81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:01:38.964373 6670 raft_consensus.cc:2258] T da54372469a14afb88d33d05a27670d5 P 19974138dea64c4586021b1acbab0e81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:01:38.964639 6672 raft_consensus.cc:2258] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:01:38.966616 6670 ts_tablet_manager.cc:1905] T da54372469a14afb88d33d05a27670d5 P 19974138dea64c4586021b1acbab0e81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 03:01:38.967114 6672 ts_tablet_manager.cc:1905] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 03:01:38 dist-test-slave-8zvk krb5kdc[6127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.193@KRBTEST.COM
I20260925 03:01:38.974260 6670 ts_tablet_manager.cc:1918] T da54372469a14afb88d33d05a27670d5 P 19974138dea64c4586021b1acbab0e81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 03:01:38.974442 6670 log.cc:1196] T da54372469a14afb88d33d05a27670d5 P 19974138dea64c4586021b1acbab0e81: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-0/wal/wals/da54372469a14afb88d33d05a27670d5
I20260925 03:01:38.974987 6672 ts_tablet_manager.cc:1918] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 03:01:38.975044 6670 ts_tablet_manager.cc:1939] T da54372469a14afb88d33d05a27670d5 P 19974138dea64c4586021b1acbab0e81: Deleting consensus metadata
I20260925 03:01:38.975224 6672 log.cc:1196] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/da54372469a14afb88d33d05a27670d5
I20260925 03:01:38.975864 6672 ts_tablet_manager.cc:1939] T da54372469a14afb88d33d05a27670d5 P f14baf89a61e474a9952fabbf3eca69f: Deleting consensus metadata
I20260925 03:01:38.976955 6157 catalog_manager.cc:4841] TS 19974138dea64c4586021b1acbab0e81 (127.18.196.193:39175): tablet da54372469a14afb88d33d05a27670d5 (table test-table [id=a45f6a75031846abaf61bf32a4f051dc]) successfully deleted
I20260925 03:01:38.977631 6156 catalog_manager.cc:4841] TS f14baf89a61e474a9952fabbf3eca69f (127.18.196.195:35687): tablet da54372469a14afb88d33d05a27670d5 (table test-table [id=a45f6a75031846abaf61bf32a4f051dc]) successfully deleted
I20260925 03:01:38.978363 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:38.940851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37090 (local address 127.18.196.193:39175)
0925 03:01:38.941087 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:38.941095 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:38.957996 (+ 16901us) server_negotiation.cc:407] Connection header received
0925 03:01:38.958497 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:38.958504 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:38.958622 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:38.958708 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:38.960943 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.962121 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.964163 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:38.964886 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:38.965007 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:38.970515 (+ 5508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:38.970531 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:38.970547 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:38.970586 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:38.974685 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:38.975704 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:38.975716 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:38.975722 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:38.975909 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:38.976392 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:38.976398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:38.976400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:38.976843 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:38.977135 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:38.977737 (+ 602us) server_negotiation.cc:299] Negotiation successful
0925 03:01:38.977983 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 03:01:38.979074 6357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:39.068324 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 6233
I20260925 03:01:39.093945 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 6368
Sep 25 03:01:39 dist-test-slave-8zvk krb5kdc[6127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.194@KRBTEST.COM
I20260925 03:01:39.110670 6644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:39.094718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58778 (local address 127.18.196.194:43941)
0925 03:01:39.094971 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:39.094979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:39.095056 (+ 77us) server_negotiation.cc:407] Connection header received
0925 03:01:39.095221 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:39.095229 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:39.095314 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:39.095430 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:39.096615 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:39.097739 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:39.099168 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:39.099781 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:39.099913 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:39.104320 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:39.104339 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:39.104359 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:39.104408 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:39.107987 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:39.108705 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:39.108714 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:39.108717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:39.108866 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:39.109204 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:39.109209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:39.109212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:39.109666 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:39.109950 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:39.110114 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 03:01:39.110317 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 03:01:39.111436 6491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:39.199999 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 6368
I20260925 03:01:39.220002 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 6501
Sep 25 03:01:39 dist-test-slave-8zvk krb5kdc[6127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.195@KRBTEST.COM
I20260925 03:01:39.239185 6645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:39.221628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46280 (local address 127.18.196.195:35687)
0925 03:01:39.222449 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:39.222456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:39.222476 (+ 20us) server_negotiation.cc:407] Connection header received
0925 03:01:39.222546 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:39.222551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:39.222637 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:39.222789 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:39.224532 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:39.225344 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:39.227367 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:39.227809 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:39.227944 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:39.233225 (+ 5281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:39.233236 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:39.233249 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:39.233278 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:39.236282 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:39.236998 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:39.237005 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:39.237007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:39.237165 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:39.237554 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:39.237558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:39.237560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:39.237890 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:39.238192 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:39.238519 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 03:01:39.238724 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690}
I20260925 03:01:39.239915 6624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:39.359342 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 6501
I20260925 03:01:39.379113 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 6140
I20260925 03:01:39.380151 6202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:39.487881 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 6140
2026-09-25T03:01:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4295 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T03:01:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:01:39Z Disabled control of system clock
I20260925 03:01:39.564448 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:41045
--webserver_interface=127.18.196.254
--webserver_port=0
--builtin_ntp_servers=127.18.196.212:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:41045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 03:01:39.879910 6681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:39.880149 6681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:39.880223 6681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:39.887187 6681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:01:39.887279 6681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:39.887326 6681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:39.887382 6681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:01:39.887424 6681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:01:39.896675 6681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:41045
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:41045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:39.897819 6681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:39.899545 6681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:39.910638 6686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:39.910715 6689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:39.910683 6687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:39.911823 6681 server_base.cc:1034] running on GCE node
I20260925 03:01:39.912603 6681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:39.913899 6681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:39.915112 6681 hybrid_clock.cc:648] HybridClock initialized: now 1790305299915068 us; error 54 us; skew 500 ppm
I20260925 03:01:39.915467 6681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:01:39.917627 6681 webserver.cc:460] Webserver started at http://127.18.196.254:42439/ using document root <none> and password file <none>
I20260925 03:01:39.918275 6681 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:39.918408 6681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:39.918718 6681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:39.921533 6681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "4253bb6a0d1c49cf9d8250adae68b38d"
format_stamp: "Formatted at 2026-09-25 03:01:39 on dist-test-slave-8zvk"
server_key: "730a69ab415bca6dd8bf43ef14d35714"
server_key_iv: "e351b3a94abaeff279bcf86cb2465c9c"
server_key_version: "encryptionkey@0"
I20260925 03:01:39.922356 6681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "4253bb6a0d1c49cf9d8250adae68b38d"
format_stamp: "Formatted at 2026-09-25 03:01:39 on dist-test-slave-8zvk"
server_key: "730a69ab415bca6dd8bf43ef14d35714"
server_key_iv: "e351b3a94abaeff279bcf86cb2465c9c"
server_key_version: "encryptionkey@0"
I20260925 03:01:39.927352 6681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 03:01:39.930691 6695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:39.931905 6681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260925 03:01:39.932083 6681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "4253bb6a0d1c49cf9d8250adae68b38d"
format_stamp: "Formatted at 2026-09-25 03:01:39 on dist-test-slave-8zvk"
server_key: "730a69ab415bca6dd8bf43ef14d35714"
server_key_iv: "e351b3a94abaeff279bcf86cb2465c9c"
server_key_version: "encryptionkey@0"
I20260925 03:01:39.932281 6681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:39.954213 6681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:39.955078 6681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:39.955348 6681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:39.972901 6681 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:41045
I20260925 03:01:39.972963 6747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:41045 every 8 connection(s)
I20260925 03:01:39.974182 6681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:01:39.975441 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 6681
I20260925 03:01:39.975633 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:01:39.976189 19219 external_mini_cluster.cc:1428] Setting key 592043816b71e047f29569c53ef97d3e
I20260925 03:01:39.978569 6748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:39.988538 6751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:39.978006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42970 (local address 127.18.196.254:41045)
0925 03:01:39.979111 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:39.979121 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:39.979159 (+ 38us) server_negotiation.cc:407] Connection header received
0925 03:01:39.980676 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:39.980695 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:39.981084 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:39.981452 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:39.982001 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:39.983348 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:39.984551 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:39.985156 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:39.985337 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:39.986445 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:39.986452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:39.986463 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:39.987062 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:39.987125 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:39.987164 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:39.987292 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:39.987489 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 03:01:39.987741 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":447,"threads_started":1}
I20260925 03:01:39.988935 6748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4253bb6a0d1c49cf9d8250adae68b38d: Bootstrap starting.
I20260925 03:01:39.992031 6748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4253bb6a0d1c49cf9d8250adae68b38d: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:39.992982 6748 log.cc:824] T 00000000000000000000000000000000 P 4253bb6a0d1c49cf9d8250adae68b38d: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:39.995471 6748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4253bb6a0d1c49cf9d8250adae68b38d: No bootstrap required, opened a new log
I20260925 03:01:40.002287 6748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4253bb6a0d1c49cf9d8250adae68b38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4253bb6a0d1c49cf9d8250adae68b38d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 41045 } }
I20260925 03:01:40.002646 6748 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4253bb6a0d1c49cf9d8250adae68b38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:40.002749 6748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4253bb6a0d1c49cf9d8250adae68b38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4253bb6a0d1c49cf9d8250adae68b38d, State: Initialized, Role: FOLLOWER
I20260925 03:01:40.003275 6748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4253bb6a0d1c49cf9d8250adae68b38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4253bb6a0d1c49cf9d8250adae68b38d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 41045 } }
I20260925 03:01:40.003513 6748 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4253bb6a0d1c49cf9d8250adae68b38d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:01:40.003654 6748 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4253bb6a0d1c49cf9d8250adae68b38d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:01:40.003824 6748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4253bb6a0d1c49cf9d8250adae68b38d [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:40.005918 6748 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4253bb6a0d1c49cf9d8250adae68b38d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4253bb6a0d1c49cf9d8250adae68b38d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 41045 } }
I20260925 03:01:40.006399 6748 leader_election.cc:302] T 00000000000000000000000000000000 P 4253bb6a0d1c49cf9d8250adae68b38d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4253bb6a0d1c49cf9d8250adae68b38d; no voters:
I20260925 03:01:40.006990 6748 leader_election.cc:288] T 00000000000000000000000000000000 P 4253bb6a0d1c49cf9d8250adae68b38d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:01:40.007128 6753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4253bb6a0d1c49cf9d8250adae68b38d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:01:40.007794 6753 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4253bb6a0d1c49cf9d8250adae68b38d [term 1 LEADER]: Becoming Leader. State: Replica: 4253bb6a0d1c49cf9d8250adae68b38d, State: Running, Role: LEADER
I20260925 03:01:40.008329 6753 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4253bb6a0d1c49cf9d8250adae68b38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4253bb6a0d1c49cf9d8250adae68b38d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 41045 } }
I20260925 03:01:40.008953 6748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4253bb6a0d1c49cf9d8250adae68b38d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:01:40.010805 6754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4253bb6a0d1c49cf9d8250adae68b38d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4253bb6a0d1c49cf9d8250adae68b38d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4253bb6a0d1c49cf9d8250adae68b38d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 41045 } } }
I20260925 03:01:40.011222 6754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4253bb6a0d1c49cf9d8250adae68b38d [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:40.011550 6755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4253bb6a0d1c49cf9d8250adae68b38d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4253bb6a0d1c49cf9d8250adae68b38d. Latest consensus state: current_term: 1 leader_uuid: "4253bb6a0d1c49cf9d8250adae68b38d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4253bb6a0d1c49cf9d8250adae68b38d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 41045 } } }
I20260925 03:01:40.011896 6755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4253bb6a0d1c49cf9d8250adae68b38d [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:40.013518 6758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:01:40.017449 6758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:01:40.026470 6758 catalog_manager.cc:1347] Generated new cluster ID: d50fb37a08d246dba8e3473128640016
I20260925 03:01:40.026588 6758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:01:40.035919 6758 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:01:40.037235 6758 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:01:40.052765 6758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4253bb6a0d1c49cf9d8250adae68b38d: Generated new TSK 0
I20260925 03:01:40.053586 6758 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:01:40.065845 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.193:0
--local_ip_for_outbound_sockets=127.18.196.193
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:41045
--builtin_ntp_servers=127.18.196.212:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 03:01:40.390890 6772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:40.391147 6772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:40.391281 6772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:40.398454 6772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:40.398563 6772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:40.398717 6772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.193
I20260925 03:01:40.407783 6772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.196.193
--webserver_port=0
--tserver_master_addrs=127.18.196.254:41045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.193
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:40.409070 6772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:40.410918 6772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:40.423237 6778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:40.423286 6777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:40.423499 6780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:40.424106 6772 server_base.cc:1034] running on GCE node
I20260925 03:01:40.424741 6772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:40.425690 6772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:40.426949 6772 hybrid_clock.cc:648] HybridClock initialized: now 1790305300426887 us; error 46 us; skew 500 ppm
I20260925 03:01:40.427280 6772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:01:40.429347 6772 webserver.cc:460] Webserver started at http://127.18.196.193:45835/ using document root <none> and password file <none>
I20260925 03:01:40.430167 6772 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:40.430279 6772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:40.430544 6772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:40.433344 6772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-0/data/instance:
uuid: "7c27192b09d1418b8e18f1b004bc2059"
format_stamp: "Formatted at 2026-09-25 03:01:40 on dist-test-slave-8zvk"
server_key: "02d76103d49576965a5a2d8c6795e667"
server_key_iv: "30eb95621d4c56ffc11c4495260b9ed2"
server_key_version: "encryptionkey@0"
I20260925 03:01:40.434094 6772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance:
uuid: "7c27192b09d1418b8e18f1b004bc2059"
format_stamp: "Formatted at 2026-09-25 03:01:40 on dist-test-slave-8zvk"
server_key: "02d76103d49576965a5a2d8c6795e667"
server_key_iv: "30eb95621d4c56ffc11c4495260b9ed2"
server_key_version: "encryptionkey@0"
I20260925 03:01:40.439200 6772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 03:01:40.442399 6786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:40.443712 6772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 03:01:40.443863 6772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-0/wal
uuid: "7c27192b09d1418b8e18f1b004bc2059"
format_stamp: "Formatted at 2026-09-25 03:01:40 on dist-test-slave-8zvk"
server_key: "02d76103d49576965a5a2d8c6795e667"
server_key_iv: "30eb95621d4c56ffc11c4495260b9ed2"
server_key_version: "encryptionkey@0"
I20260925 03:01:40.444037 6772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:40.472728 6772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:40.473449 6772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:40.473688 6772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:40.474548 6772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:01:40.475937 6772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:01:40.476029 6772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:40.476109 6772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:01:40.476166 6772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:40.508366 6772 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.193:33851
I20260925 03:01:40.508448 6899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.193:33851 every 8 connection(s)
I20260925 03:01:40.509740 6772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-0/data/info.pb
I20260925 03:01:40.510466 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 6772
I20260925 03:01:40.510666 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-0/wal/instance
I20260925 03:01:40.511283 19219 external_mini_cluster.cc:1428] Setting key 28fd4b29febf5cbc707007a64dbfcc4d
I20260925 03:01:40.514433 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.194:0
--local_ip_for_outbound_sockets=127.18.196.194
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:41045
--builtin_ntp_servers=127.18.196.212:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 03:01:40.522531 6902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:40.512478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:33969 (local address 127.18.196.254:41045)
0925 03:01:40.512957 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:40.512964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:40.514225 (+ 1261us) server_negotiation.cc:407] Connection header received
0925 03:01:40.516088 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:40.516094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:40.516186 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:40.516305 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:40.517332 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:40.518103 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:40.519818 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:40.520249 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:40.520381 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:40.521680 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:40.521686 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:40.521691 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:40.521716 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:40.521765 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:40.521778 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:40.521877 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:40.522090 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 03:01:40.522188 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":369,"threads_started":1}
I20260925 03:01:40.523757 6903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:40.513171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:41045 (local address 127.18.196.193:33969)
0925 03:01:40.514085 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:40.514126 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:40.515827 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:40.516535 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:40.516543 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:40.516627 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:01:40.517170 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:40.517186 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:40.518279 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:40.518288 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:40.519670 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:40.519682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:40.520470 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:40.520478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:40.520720 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:40.521390 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 03:01:40.521392 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:01:40.521430 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 03:01:40.521501 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:01:40.521876 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:01:40.521889 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:01:40.521899 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 03:01:40.522078 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 03:01:40.522431 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":297,"threads_started":1}
I20260925 03:01:40.525183 6900 heartbeater.cc:347] Connected to a master server at 127.18.196.254:41045
I20260925 03:01:40.525477 6900 heartbeater.cc:464] Registering TS with master...
I20260925 03:01:40.526333 6900 heartbeater.cc:511] Master 127.18.196.254:41045 requested a full tablet report, sending...
I20260925 03:01:40.528326 6712 ts_manager.cc:194] Registered new tserver with Master: 7c27192b09d1418b8e18f1b004bc2059 (127.18.196.193:33851)
I20260925 03:01:40.529727 6712 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.196.193:33969
W20260925 03:01:40.834781 6904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:40.835147 6904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:40.835301 6904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:40.842463 6904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:40.842582 6904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:40.842741 6904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.194
I20260925 03:01:40.851508 6904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.196.194
--webserver_port=0
--tserver_master_addrs=127.18.196.254:41045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.194
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:40.852675 6904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:40.854341 6904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:40.866940 6909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:40.866956 6910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:40.867009 6912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:40.867720 6904 server_base.cc:1034] running on GCE node
I20260925 03:01:40.868275 6904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:40.869153 6904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:40.870342 6904 hybrid_clock.cc:648] HybridClock initialized: now 1790305300870304 us; error 28 us; skew 500 ppm
I20260925 03:01:40.870638 6904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:01:40.872746 6904 webserver.cc:460] Webserver started at http://127.18.196.194:45933/ using document root <none> and password file <none>
I20260925 03:01:40.873390 6904 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:40.873502 6904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:40.873805 6904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:40.876581 6904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-1/data/instance:
uuid: "6256c465eae64e13a2d192ecce994605"
format_stamp: "Formatted at 2026-09-25 03:01:40 on dist-test-slave-8zvk"
server_key: "79f029f81b4397f85ce0009d6970e292"
server_key_iv: "d4b4fca91a9dc89ea745804d781ced89"
server_key_version: "encryptionkey@0"
I20260925 03:01:40.877354 6904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance:
uuid: "6256c465eae64e13a2d192ecce994605"
format_stamp: "Formatted at 2026-09-25 03:01:40 on dist-test-slave-8zvk"
server_key: "79f029f81b4397f85ce0009d6970e292"
server_key_iv: "d4b4fca91a9dc89ea745804d781ced89"
server_key_version: "encryptionkey@0"
I20260925 03:01:40.882292 6904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 03:01:40.885628 6918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:40.886931 6904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 03:01:40.887092 6904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-1/wal
uuid: "6256c465eae64e13a2d192ecce994605"
format_stamp: "Formatted at 2026-09-25 03:01:40 on dist-test-slave-8zvk"
server_key: "79f029f81b4397f85ce0009d6970e292"
server_key_iv: "d4b4fca91a9dc89ea745804d781ced89"
server_key_version: "encryptionkey@0"
I20260925 03:01:40.887300 6904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:40.902166 6904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:40.903059 6904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:40.903352 6904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:40.904274 6904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:01:40.905660 6904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:01:40.905782 6904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:40.905905 6904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:01:40.905997 6904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:40.938647 6904 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.194:39697
I20260925 03:01:40.938727 7031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.194:39697 every 8 connection(s)
I20260925 03:01:40.940083 6904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-1/data/info.pb
I20260925 03:01:40.940912 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 6904
I20260925 03:01:40.941102 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-1/wal/instance
I20260925 03:01:40.941623 19219 external_mini_cluster.cc:1428] Setting key 53da03d23169bdd276ca2ab7435ac8b8
I20260925 03:01:40.944944 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.196.195:0
--local_ip_for_outbound_sockets=127.18.196.195
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:41045
--builtin_ntp_servers=127.18.196.212:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 03:01:40.954830 6902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:40.942872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.194:53919 (local address 127.18.196.254:41045)
0925 03:01:40.943067 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:40.943074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:40.944675 (+ 1601us) server_negotiation.cc:407] Connection header received
0925 03:01:40.947549 (+ 2874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:40.947554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:40.947620 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:40.947703 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:40.949015 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:40.949699 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:40.952141 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:40.952522 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:40.952616 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:40.954146 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:40.954151 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:40.954154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:40.954172 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:40.954203 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:40.954214 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:40.954283 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:40.954535 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 03:01:40.954608 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 03:01:40.955822 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:40.943424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:41045 (local address 127.18.196.194:53919)
0925 03:01:40.944518 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:40.944566 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:40.947283 (+ 2717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:40.947955 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:40.947968 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:40.948077 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:01:40.948858 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:40.948883 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:40.949852 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:40.949860 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:40.952007 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:40.952025 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:40.952731 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:40.952740 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:40.953080 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:40.953809 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 03:01:40.953815 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:01:40.953865 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 03:01:40.953959 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:01:40.954313 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:01:40.954326 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:01:40.954336 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 03:01:40.954501 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 03:01:40.954915 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":350,"threads_started":1}
I20260925 03:01:40.957167 7032 heartbeater.cc:347] Connected to a master server at 127.18.196.254:41045
I20260925 03:01:40.957440 7032 heartbeater.cc:464] Registering TS with master...
I20260925 03:01:40.958096 7032 heartbeater.cc:511] Master 127.18.196.254:41045 requested a full tablet report, sending...
I20260925 03:01:40.959486 6712 ts_manager.cc:194] Registered new tserver with Master: 6256c465eae64e13a2d192ecce994605 (127.18.196.194:39697)
I20260925 03:01:40.960270 6712 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.196.194:53919
W20260925 03:01:41.269953 7035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:41.270238 7035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:41.270393 7035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:41.277796 7035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:41.277940 7035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:41.278095 7035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.196.195
I20260925 03:01:41.286924 7035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.196.195
--webserver_port=0
--tserver_master_addrs=127.18.196.254:41045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.196.195
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:41.288064 7035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:41.289752 7035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:41.302577 7040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:41.302510 7041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:41.302546 7043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:41.303602 7035 server_base.cc:1034] running on GCE node
I20260925 03:01:41.304177 7035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:41.304993 7035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:41.306182 7035 hybrid_clock.cc:648] HybridClock initialized: now 1790305301306138 us; error 34 us; skew 500 ppm
I20260925 03:01:41.306438 7035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 03:01:41.308640 7035 webserver.cc:460] Webserver started at http://127.18.196.195:35573/ using document root <none> and password file <none>
I20260925 03:01:41.309242 7035 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:41.309317 7035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:41.309587 7035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:41.312201 7035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-2/data/instance:
uuid: "dc43ba95d2314b63acc9a8fd20421016"
format_stamp: "Formatted at 2026-09-25 03:01:41 on dist-test-slave-8zvk"
server_key: "6a908736417ab4440cc1039ce5a6cffe"
server_key_iv: "44da1c875cc29f7b72f831349ec1b772"
server_key_version: "encryptionkey@0"
I20260925 03:01:41.312897 7035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance:
uuid: "dc43ba95d2314b63acc9a8fd20421016"
format_stamp: "Formatted at 2026-09-25 03:01:41 on dist-test-slave-8zvk"
server_key: "6a908736417ab4440cc1039ce5a6cffe"
server_key_iv: "44da1c875cc29f7b72f831349ec1b772"
server_key_version: "encryptionkey@0"
I20260925 03:01:41.317387 7035 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 03:01:41.320710 7049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:41.321976 7035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 03:01:41.322099 7035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-2/wal
uuid: "dc43ba95d2314b63acc9a8fd20421016"
format_stamp: "Formatted at 2026-09-25 03:01:41 on dist-test-slave-8zvk"
server_key: "6a908736417ab4440cc1039ce5a6cffe"
server_key_iv: "44da1c875cc29f7b72f831349ec1b772"
server_key_version: "encryptionkey@0"
I20260925 03:01:41.322259 7035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:41.363459 7035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:41.364241 7035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:41.364524 7035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:41.365468 7035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 03:01:41.366822 7035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 03:01:41.366995 7035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:41.367118 7035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 03:01:41.367213 7035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:41.399559 7035 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.195:46405
I20260925 03:01:41.399607 7161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.195:46405 every 8 connection(s)
I20260925 03:01:41.400853 7035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-2/data/info.pb
I20260925 03:01:41.403719 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 7035
I20260925 03:01:41.403903 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-2/wal/instance
I20260925 03:01:41.404321 19219 external_mini_cluster.cc:1428] Setting key 40baad1c6b509e6e26eb29b6cf8ce5d4
I20260925 03:01:41.413087 6902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:41.403389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.195:48931 (local address 127.18.196.254:41045)
0925 03:01:41.403615 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:41.403622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:41.405159 (+ 1537us) server_negotiation.cc:407] Connection header received
0925 03:01:41.407220 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:41.407226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:41.407311 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:41.407391 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:41.408368 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.409158 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.410649 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.411084 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.411185 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:41.412438 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:41.412442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:41.412445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:41.412461 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:41.412489 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:41.412500 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:41.412565 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:41.412771 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 03:01:41.412848 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 03:01:41.414057 7164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:41.403961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:41045 (local address 127.18.196.195:48931)
0925 03:01:41.404992 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:41.405042 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:41.406993 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:41.407574 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:41.407582 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:41.407667 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:01:41.408202 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:41.408230 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.409284 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.409288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:41.410538 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:41.410550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.411262 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.411267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:41.411469 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:41.412174 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 03:01:41.412177 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:01:41.412213 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 03:01:41.412287 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:01:41.412578 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:01:41.412585 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:01:41.412590 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 03:01:41.412719 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 03:01:41.412994 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":335,"threads_started":1}
I20260925 03:01:41.415313 7162 heartbeater.cc:347] Connected to a master server at 127.18.196.254:41045
I20260925 03:01:41.415560 7162 heartbeater.cc:464] Registering TS with master...
I20260925 03:01:41.416216 7162 heartbeater.cc:511] Master 127.18.196.254:41045 requested a full tablet report, sending...
I20260925 03:01:41.417433 6712 ts_manager.cc:194] Registered new tserver with Master: dc43ba95d2314b63acc9a8fd20421016 (127.18.196.195:46405)
I20260925 03:01:41.418253 6712 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.196.195:48931
I20260925 03:01:41.419294 19219 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 03:01:41.428905 6902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:41.423170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49634 (local address 127.18.196.254:41045)
0925 03:01:41.423321 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:41.423326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:41.423788 (+ 462us) server_negotiation.cc:407] Connection header received
0925 03:01:41.423979 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:41.423985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:41.424055 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:41.424162 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:41.424635 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.425361 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.426863 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.427265 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.427360 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:41.428326 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:41.428330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:41.428333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:41.428350 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:41.428383 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:41.428397 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:41.428474 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:41.428586 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 03:01:41.428660 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 03:01:41.432809 6712 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 03:01:41.439558 6712 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 03:01:41.476706 7175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:41.450726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:46405 (local address 127.0.0.1:43374)
0925 03:01:41.461851 (+ 11125us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:41.461903 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:41.462112 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:41.468597 (+ 6485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:41.468604 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:41.468633 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:41.469346 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:41.469363 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.471359 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.471363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:41.474226 (+ 2863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:41.474242 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.475633 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.475640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:41.475853 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:41.475909 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 03:01:41.476072 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 03:01:41.476282 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10824,"mutex_wait_us":296,"thread_start_us":17536,"threads_started":1}
I20260925 03:01:41.476899 7176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:41.451066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43374 (local address 127.18.196.195:46405)
0925 03:01:41.467491 (+ 16425us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:41.467504 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:41.467550 (+ 46us) server_negotiation.cc:407] Connection header received
0925 03:01:41.467696 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:41.467708 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:41.468015 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:41.468601 (+ 586us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:41.469571 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.471242 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.474485 (+ 3243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.475440 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.475634 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:41.475858 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:41.476367 (+ 509us) server_negotiation.cc:299] Negotiation successful
0925 03:01:41.476531 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16183,"thread_start_us":15989,"threads_started":1}
I20260925 03:01:41.477200 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:41.453057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52388 (local address 127.18.196.193:33851)
0925 03:01:41.467755 (+ 14698us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:41.467767 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:41.468626 (+ 859us) server_negotiation.cc:407] Connection header received
0925 03:01:41.468915 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:41.468922 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:41.469139 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:41.469302 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:41.470232 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.471654 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.474050 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.475459 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.475617 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:41.475859 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:41.476660 (+ 801us) server_negotiation.cc:299] Negotiation successful
0925 03:01:41.476840 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14523,"thread_start_us":14416,"threads_started":1}
I20260925 03:01:41.477478 7174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:41.452452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33851 (local address 127.0.0.1:52388)
0925 03:01:41.468518 (+ 16066us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:41.468535 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:41.468727 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:41.469594 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:41.469603 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:41.469636 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:41.470065 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:41.470079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.471800 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.471809 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:41.473840 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:41.473857 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.476006 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.476014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:41.476199 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:41.476229 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 03:01:41.476320 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 03:01:41.476450 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15943,"mutex_wait_us":8,"thread_start_us":15484,"threads_started":1}
I20260925 03:01:41.477599 7177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:41.451369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49428 (local address 127.18.196.194:39697)
0925 03:01:41.468248 (+ 16879us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:41.468259 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:41.468579 (+ 320us) server_negotiation.cc:407] Connection header received
0925 03:01:41.468701 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:41.468710 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:41.468936 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:41.469901 (+ 965us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:41.470120 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.471164 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.474459 (+ 3295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.475882 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.476043 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:41.476227 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:41.476978 (+ 751us) server_negotiation.cc:299] Negotiation successful
0925 03:01:41.477238 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16692,"thread_start_us":16866,"threads_started":1}
I20260925 03:01:41.476852 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:41.450886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:39697 (local address 127.0.0.1:49428)
0925 03:01:41.468241 (+ 17355us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:41.468264 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:41.468417 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:41.469148 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:41.469153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:41.469184 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:41.469661 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:41.469676 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.471289 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.471296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:41.474226 (+ 2930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:41.474242 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.476110 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.476116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:41.476271 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:41.476298 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 03:01:41.476383 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 03:01:41.476508 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17181,"mutex_wait_us":291,"thread_start_us":10457,"threads_started":1}
I20260925 03:01:41.481268 6966 tablet_service.cc:1467] Processing CreateTablet for tablet 8850b9ad3de34e23b92816f8179614a8 (DEFAULT_TABLE table=test-table [id=6fcbc8555bf94fe295961665ef677b0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 03:01:41.481320 6835 tablet_service.cc:1467] Processing CreateTablet for tablet 8850b9ad3de34e23b92816f8179614a8 (DEFAULT_TABLE table=test-table [id=6fcbc8555bf94fe295961665ef677b0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 03:01:41.481325 7097 tablet_service.cc:1467] Processing CreateTablet for tablet 8850b9ad3de34e23b92816f8179614a8 (DEFAULT_TABLE table=test-table [id=6fcbc8555bf94fe295961665ef677b0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 03:01:41.482527 6966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8850b9ad3de34e23b92816f8179614a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:41.482694 6835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8850b9ad3de34e23b92816f8179614a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:41.482694 7097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8850b9ad3de34e23b92816f8179614a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:41.494114 7180 tablet_bootstrap.cc:492] T 8850b9ad3de34e23b92816f8179614a8 P 6256c465eae64e13a2d192ecce994605: Bootstrap starting.
I20260925 03:01:41.497823 7180 tablet_bootstrap.cc:654] T 8850b9ad3de34e23b92816f8179614a8 P 6256c465eae64e13a2d192ecce994605: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:41.498921 7180 log.cc:824] T 8850b9ad3de34e23b92816f8179614a8 P 6256c465eae64e13a2d192ecce994605: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:41.499859 7182 tablet_bootstrap.cc:492] T 8850b9ad3de34e23b92816f8179614a8 P dc43ba95d2314b63acc9a8fd20421016: Bootstrap starting.
I20260925 03:01:41.500419 7181 tablet_bootstrap.cc:492] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059: Bootstrap starting.
I20260925 03:01:41.503515 7181 tablet_bootstrap.cc:654] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:41.504503 7181 log.cc:824] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:41.504781 7182 tablet_bootstrap.cc:654] T 8850b9ad3de34e23b92816f8179614a8 P dc43ba95d2314b63acc9a8fd20421016: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:41.506069 7182 log.cc:824] T 8850b9ad3de34e23b92816f8179614a8 P dc43ba95d2314b63acc9a8fd20421016: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:41.507463 7180 tablet_bootstrap.cc:492] T 8850b9ad3de34e23b92816f8179614a8 P 6256c465eae64e13a2d192ecce994605: No bootstrap required, opened a new log
I20260925 03:01:41.507573 7181 tablet_bootstrap.cc:492] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059: No bootstrap required, opened a new log
I20260925 03:01:41.507766 7180 ts_tablet_manager.cc:1397] T 8850b9ad3de34e23b92816f8179614a8 P 6256c465eae64e13a2d192ecce994605: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260925 03:01:41.507833 7181 ts_tablet_manager.cc:1397] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 03:01:41.509899 7182 tablet_bootstrap.cc:492] T 8850b9ad3de34e23b92816f8179614a8 P dc43ba95d2314b63acc9a8fd20421016: No bootstrap required, opened a new log
I20260925 03:01:41.510286 7182 ts_tablet_manager.cc:1397] T 8850b9ad3de34e23b92816f8179614a8 P dc43ba95d2314b63acc9a8fd20421016: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260925 03:01:41.518415 7180 raft_consensus.cc:348] T 8850b9ad3de34e23b92816f8179614a8 P 6256c465eae64e13a2d192ecce994605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc43ba95d2314b63acc9a8fd20421016" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 46405 } } peers { permanent_uuid: "6256c465eae64e13a2d192ecce994605" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39697 } } peers { permanent_uuid: "7c27192b09d1418b8e18f1b004bc2059" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33851 } }
I20260925 03:01:41.519012 7180 raft_consensus.cc:374] T 8850b9ad3de34e23b92816f8179614a8 P 6256c465eae64e13a2d192ecce994605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:41.519153 7180 raft_consensus.cc:729] T 8850b9ad3de34e23b92816f8179614a8 P 6256c465eae64e13a2d192ecce994605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6256c465eae64e13a2d192ecce994605, State: Initialized, Role: FOLLOWER
I20260925 03:01:41.519833 7180 consensus_queue.cc:260] T 8850b9ad3de34e23b92816f8179614a8 P 6256c465eae64e13a2d192ecce994605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc43ba95d2314b63acc9a8fd20421016" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 46405 } } peers { permanent_uuid: "6256c465eae64e13a2d192ecce994605" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39697 } } peers { permanent_uuid: "7c27192b09d1418b8e18f1b004bc2059" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33851 } }
I20260925 03:01:41.519948 7182 raft_consensus.cc:348] T 8850b9ad3de34e23b92816f8179614a8 P dc43ba95d2314b63acc9a8fd20421016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc43ba95d2314b63acc9a8fd20421016" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 46405 } } peers { permanent_uuid: "6256c465eae64e13a2d192ecce994605" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39697 } } peers { permanent_uuid: "7c27192b09d1418b8e18f1b004bc2059" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33851 } }
I20260925 03:01:41.520382 7182 raft_consensus.cc:374] T 8850b9ad3de34e23b92816f8179614a8 P dc43ba95d2314b63acc9a8fd20421016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:41.520154 7181 raft_consensus.cc:348] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc43ba95d2314b63acc9a8fd20421016" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 46405 } } peers { permanent_uuid: "6256c465eae64e13a2d192ecce994605" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39697 } } peers { permanent_uuid: "7c27192b09d1418b8e18f1b004bc2059" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33851 } }
I20260925 03:01:41.520534 7182 raft_consensus.cc:729] T 8850b9ad3de34e23b92816f8179614a8 P dc43ba95d2314b63acc9a8fd20421016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc43ba95d2314b63acc9a8fd20421016, State: Initialized, Role: FOLLOWER
I20260925 03:01:41.520645 7181 raft_consensus.cc:374] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:41.520807 7181 raft_consensus.cc:729] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c27192b09d1418b8e18f1b004bc2059, State: Initialized, Role: FOLLOWER
I20260925 03:01:41.521315 7182 consensus_queue.cc:260] T 8850b9ad3de34e23b92816f8179614a8 P dc43ba95d2314b63acc9a8fd20421016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc43ba95d2314b63acc9a8fd20421016" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 46405 } } peers { permanent_uuid: "6256c465eae64e13a2d192ecce994605" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39697 } } peers { permanent_uuid: "7c27192b09d1418b8e18f1b004bc2059" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33851 } }
I20260925 03:01:41.521498 7181 consensus_queue.cc:260] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc43ba95d2314b63acc9a8fd20421016" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 46405 } } peers { permanent_uuid: "6256c465eae64e13a2d192ecce994605" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39697 } } peers { permanent_uuid: "7c27192b09d1418b8e18f1b004bc2059" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33851 } }
I20260925 03:01:41.529389 7182 ts_tablet_manager.cc:1428] T 8850b9ad3de34e23b92816f8179614a8 P dc43ba95d2314b63acc9a8fd20421016: Time spent starting tablet: real 0.019s user 0.017s sys 0.000s
I20260925 03:01:41.529824 7162 heartbeater.cc:503] Master 127.18.196.254:41045 was elected leader, sending a full tablet report...
I20260925 03:01:41.531569 7032 heartbeater.cc:503] Master 127.18.196.254:41045 was elected leader, sending a full tablet report...
I20260925 03:01:41.532497 6900 heartbeater.cc:503] Master 127.18.196.254:41045 was elected leader, sending a full tablet report...
I20260925 03:01:41.532998 7181 ts_tablet_manager.cc:1428] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059: Time spent starting tablet: real 0.025s user 0.012s sys 0.011s
I20260925 03:01:41.532440 7180 ts_tablet_manager.cc:1428] T 8850b9ad3de34e23b92816f8179614a8 P 6256c465eae64e13a2d192ecce994605: Time spent starting tablet: real 0.025s user 0.013s sys 0.011s
I20260925 03:01:41.616914 7188 raft_consensus.cc:482] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 03:01:41.617254 7188 raft_consensus.cc:504] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc43ba95d2314b63acc9a8fd20421016" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 46405 } } peers { permanent_uuid: "6256c465eae64e13a2d192ecce994605" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39697 } } peers { permanent_uuid: "7c27192b09d1418b8e18f1b004bc2059" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33851 } }
I20260925 03:01:41.619112 7188 leader_election.cc:288] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc43ba95d2314b63acc9a8fd20421016 (127.18.196.195:46405), 6256c465eae64e13a2d192ecce994605 (127.18.196.194:39697)
I20260925 03:01:41.626017 7177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:41.619727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:37207 (local address 127.18.196.194:39697)
0925 03:01:41.619915 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:41.619921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:41.620588 (+ 667us) server_negotiation.cc:407] Connection header received
0925 03:01:41.620747 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:41.620752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:41.620862 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:41.620960 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:41.621658 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.622437 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.624237 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.625015 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.625132 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:41.625181 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:41.625568 (+ 387us) server_negotiation.cc:299] Negotiation successful
0925 03:01:41.625696 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 03:01:41.626084 7190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:41.619547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.194:39697 (local address 127.18.196.193:37207)
0925 03:01:41.620461 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:41.620484 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:41.620614 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:41.620990 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:41.620994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:41.621013 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:41.621460 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:41.621473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.622577 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.622581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:41.624045 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:41.624057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.625188 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.625195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:41.625372 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:41.625419 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 03:01:41.625572 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 03:01:41.625684 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":6,"thread_start_us":528,"threads_started":1}
I20260925 03:01:41.626235 7176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:41.619382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.196.193:60825 (local address 127.18.196.195:46405)
0925 03:01:41.619611 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:41.619619 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:41.620177 (+ 558us) server_negotiation.cc:407] Connection header received
0925 03:01:41.620446 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:41.620453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:41.620568 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:41.620660 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 03:01:41.621272 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.622154 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.624275 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.625316 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.625498 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:41.625576 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:41.625846 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 03:01:41.625982 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 03:01:41.626364 7189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:41.619285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.195:46405 (local address 127.18.196.193:60825)
0925 03:01:41.620055 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:41.620082 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 03:01:41.620256 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:01:41.620705 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:01:41.620709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:01:41.620737 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 03:01:41.621100 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:41.621116 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.622278 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.622283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:41.624046 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:01:41.624058 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.625523 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.625531 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:01:41.625688 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:41.625717 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 03:01:41.625816 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 03:01:41.625939 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":117,"thread_start_us":516,"threads_started":1}
I20260925 03:01:41.627367 7117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8850b9ad3de34e23b92816f8179614a8" candidate_uuid: "7c27192b09d1418b8e18f1b004bc2059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc43ba95d2314b63acc9a8fd20421016" is_pre_election: true
I20260925 03:01:41.627444 6986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8850b9ad3de34e23b92816f8179614a8" candidate_uuid: "7c27192b09d1418b8e18f1b004bc2059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6256c465eae64e13a2d192ecce994605" is_pre_election: true
I20260925 03:01:41.627885 7117 raft_consensus.cc:2454] T 8850b9ad3de34e23b92816f8179614a8 P dc43ba95d2314b63acc9a8fd20421016 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c27192b09d1418b8e18f1b004bc2059 in term 0.
I20260925 03:01:41.627902 6986 raft_consensus.cc:2454] T 8850b9ad3de34e23b92816f8179614a8 P 6256c465eae64e13a2d192ecce994605 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c27192b09d1418b8e18f1b004bc2059 in term 0.
I20260925 03:01:41.628542 6790 leader_election.cc:302] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c27192b09d1418b8e18f1b004bc2059, dc43ba95d2314b63acc9a8fd20421016; no voters:
I20260925 03:01:41.628937 7188 raft_consensus.cc:2781] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 03:01:41.629091 7188 raft_consensus.cc:482] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 03:01:41.629204 7188 raft_consensus.cc:3037] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:41.631412 7188 raft_consensus.cc:504] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc43ba95d2314b63acc9a8fd20421016" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 46405 } } peers { permanent_uuid: "6256c465eae64e13a2d192ecce994605" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39697 } } peers { permanent_uuid: "7c27192b09d1418b8e18f1b004bc2059" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33851 } }
I20260925 03:01:41.632054 7188 leader_election.cc:288] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059 [CANDIDATE]: Term 1 election: Requested vote from peers dc43ba95d2314b63acc9a8fd20421016 (127.18.196.195:46405), 6256c465eae64e13a2d192ecce994605 (127.18.196.194:39697)
I20260925 03:01:41.632407 7117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8850b9ad3de34e23b92816f8179614a8" candidate_uuid: "7c27192b09d1418b8e18f1b004bc2059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc43ba95d2314b63acc9a8fd20421016"
I20260925 03:01:41.632505 6986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8850b9ad3de34e23b92816f8179614a8" candidate_uuid: "7c27192b09d1418b8e18f1b004bc2059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6256c465eae64e13a2d192ecce994605"
I20260925 03:01:41.632620 7117 raft_consensus.cc:3037] T 8850b9ad3de34e23b92816f8179614a8 P dc43ba95d2314b63acc9a8fd20421016 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:41.632687 6986 raft_consensus.cc:3037] T 8850b9ad3de34e23b92816f8179614a8 P 6256c465eae64e13a2d192ecce994605 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:41.635842 6986 raft_consensus.cc:2454] T 8850b9ad3de34e23b92816f8179614a8 P 6256c465eae64e13a2d192ecce994605 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c27192b09d1418b8e18f1b004bc2059 in term 1.
I20260925 03:01:41.635869 7117 raft_consensus.cc:2454] T 8850b9ad3de34e23b92816f8179614a8 P dc43ba95d2314b63acc9a8fd20421016 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c27192b09d1418b8e18f1b004bc2059 in term 1.
I20260925 03:01:41.636206 6790 leader_election.cc:302] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c27192b09d1418b8e18f1b004bc2059, dc43ba95d2314b63acc9a8fd20421016; no voters:
I20260925 03:01:41.636487 7188 raft_consensus.cc:2781] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:01:41.636986 7188 raft_consensus.cc:686] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059 [term 1 LEADER]: Becoming Leader. State: Replica: 7c27192b09d1418b8e18f1b004bc2059, State: Running, Role: LEADER
I20260925 03:01:41.637413 7188 consensus_queue.cc:237] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc43ba95d2314b63acc9a8fd20421016" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 46405 } } peers { permanent_uuid: "6256c465eae64e13a2d192ecce994605" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39697 } } peers { permanent_uuid: "7c27192b09d1418b8e18f1b004bc2059" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33851 } }
I20260925 03:01:41.641606 6712 catalog_manager.cc:5495] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c27192b09d1418b8e18f1b004bc2059 (127.18.196.193). New cstate: current_term: 1 leader_uuid: "7c27192b09d1418b8e18f1b004bc2059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc43ba95d2314b63acc9a8fd20421016" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 46405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6256c465eae64e13a2d192ecce994605" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c27192b09d1418b8e18f1b004bc2059" member_type: VOTER last_known_addr { host: "127.18.196.193" port: 33851 } health_report { overall_health: HEALTHY } } }
W20260925 03:01:41.652561 7163 tablet.cc:2369] T 8850b9ad3de34e23b92816f8179614a8 P dc43ba95d2314b63acc9a8fd20421016: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 03:01:41.663319 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:41.657856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52398 (local address 127.18.196.193:33851)
0925 03:01:41.658065 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:41.658072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:41.658157 (+ 85us) server_negotiation.cc:407] Connection header received
0925 03:01:41.658414 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:41.658418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:41.658504 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:41.658650 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:01:41.659098 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.659797 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.661159 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.661570 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.661703 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:41.662208 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:01:41.662692 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:01:41.662876 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:41.662964 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 03:01:41.663052 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 03:01:41.671227 7117 raft_consensus.cc:1261] T 8850b9ad3de34e23b92816f8179614a8 P dc43ba95d2314b63acc9a8fd20421016 [term 1 FOLLOWER]: Refusing update from remote peer 7c27192b09d1418b8e18f1b004bc2059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:01:41.671224 6986 raft_consensus.cc:1261] T 8850b9ad3de34e23b92816f8179614a8 P 6256c465eae64e13a2d192ecce994605 [term 1 FOLLOWER]: Refusing update from remote peer 7c27192b09d1418b8e18f1b004bc2059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:01:41.672148 7191 consensus_queue.cc:1035] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6256c465eae64e13a2d192ecce994605" member_type: VOTER last_known_addr { host: "127.18.196.194" port: 39697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:01:41.672534 7188 consensus_queue.cc:1035] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc43ba95d2314b63acc9a8fd20421016" member_type: VOTER last_known_addr { host: "127.18.196.195" port: 46405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:01:41.683446 7194 mvcc.cc:205] Tried to move back new op lower bound from 7333090515635343360 to 7333090515510358016. Current Snapshot: MvccSnapshot[applied={T|T < 7333090515635343360}]
I20260925 03:01:41.700235 6712 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 03:01:41.700529 6712 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 03:01:41.704031 6712 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4253bb6a0d1c49cf9d8250adae68b38d: Sending DeleteTablet for 3 replicas of tablet 8850b9ad3de34e23b92816f8179614a8
I20260925 03:01:41.705202 7097 tablet_service.cc:1514] Processing DeleteTablet for tablet 8850b9ad3de34e23b92816f8179614a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 03:01:41 UTC) from {username='slave'} at 127.0.0.1:43374
I20260925 03:01:41.705652 6966 tablet_service.cc:1514] Processing DeleteTablet for tablet 8850b9ad3de34e23b92816f8179614a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 03:01:41 UTC) from {username='slave'} at 127.0.0.1:49428
I20260925 03:01:41.705684 6835 tablet_service.cc:1514] Processing DeleteTablet for tablet 8850b9ad3de34e23b92816f8179614a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 03:01:41 UTC) from {username='slave'} at 127.0.0.1:52388
I20260925 03:01:41.707108 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 6772
I20260925 03:01:41.718717 7205 tablet_replica.cc:330] T 8850b9ad3de34e23b92816f8179614a8 P 6256c465eae64e13a2d192ecce994605: stopping tablet replica
I20260925 03:01:41.719332 7205 raft_consensus.cc:2229] T 8850b9ad3de34e23b92816f8179614a8 P 6256c465eae64e13a2d192ecce994605 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:01:41.719990 7205 raft_consensus.cc:2258] T 8850b9ad3de34e23b92816f8179614a8 P 6256c465eae64e13a2d192ecce994605 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:01:41.720947 7206 tablet_replica.cc:330] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059: stopping tablet replica
I20260925 03:01:41.721428 7206 raft_consensus.cc:2229] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059 [term 1 LEADER]: Raft consensus shutting down.
I20260925 03:01:41.722250 7206 raft_consensus.cc:2258] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:01:41.724211 7204 tablet_replica.cc:330] T 8850b9ad3de34e23b92816f8179614a8 P dc43ba95d2314b63acc9a8fd20421016: stopping tablet replica
I20260925 03:01:41.724761 7204 raft_consensus.cc:2229] T 8850b9ad3de34e23b92816f8179614a8 P dc43ba95d2314b63acc9a8fd20421016 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 03:01:41.725250 7204 raft_consensus.cc:2258] T 8850b9ad3de34e23b92816f8179614a8 P dc43ba95d2314b63acc9a8fd20421016 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 03:01:41.725819 7205 ts_tablet_manager.cc:1905] T 8850b9ad3de34e23b92816f8179614a8 P 6256c465eae64e13a2d192ecce994605: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 03:01:41.726671 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:41.710150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52414 (local address 127.18.196.193:33851)
0925 03:01:41.710556 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:41.710562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:41.714158 (+ 3596us) server_negotiation.cc:407] Connection header received
0925 03:01:41.714385 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:41.714391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:41.714488 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:41.714613 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:41.719090 (+ 4477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.720379 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.722538 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.723226 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.723327 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:41.724778 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:41.724799 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:41.724815 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:41.725652 (+ 837us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:41.725722 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:41.725750 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:41.725844 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:41.726177 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 03:01:41.726279 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287}
I20260925 03:01:41.726934 7206 ts_tablet_manager.cc:1905] T 8850b9ad3de34e23b92816f8179614a8 P 7c27192b09d1418b8e18f1b004bc2059: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 03:01:41.727931 6895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:41.729797 7204 ts_tablet_manager.cc:1905] T 8850b9ad3de34e23b92816f8179614a8 P dc43ba95d2314b63acc9a8fd20421016: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 03:01:41.733217 7205 ts_tablet_manager.cc:1918] T 8850b9ad3de34e23b92816f8179614a8 P 6256c465eae64e13a2d192ecce994605: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 03:01:41.733400 7205 log.cc:1196] T 8850b9ad3de34e23b92816f8179614a8 P 6256c465eae64e13a2d192ecce994605: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-1/wal/wals/8850b9ad3de34e23b92816f8179614a8
I20260925 03:01:41.733870 7205 ts_tablet_manager.cc:1939] T 8850b9ad3de34e23b92816f8179614a8 P 6256c465eae64e13a2d192ecce994605: Deleting consensus metadata
I20260925 03:01:41.735486 6697 catalog_manager.cc:4841] TS 6256c465eae64e13a2d192ecce994605 (127.18.196.194:39697): tablet 8850b9ad3de34e23b92816f8179614a8 (table test-table [id=6fcbc8555bf94fe295961665ef677b0b]) successfully deleted
I20260925 03:01:41.737044 7204 ts_tablet_manager.cc:1918] T 8850b9ad3de34e23b92816f8179614a8 P dc43ba95d2314b63acc9a8fd20421016: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 03:01:41.737202 7204 log.cc:1196] T 8850b9ad3de34e23b92816f8179614a8 P dc43ba95d2314b63acc9a8fd20421016: Deleting WAL directory at /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304987644417-19219-0/minicluster-data/ts-2/wal/wals/8850b9ad3de34e23b92816f8179614a8
I20260925 03:01:41.737736 7204 ts_tablet_manager.cc:1939] T 8850b9ad3de34e23b92816f8179614a8 P dc43ba95d2314b63acc9a8fd20421016: Deleting consensus metadata
I20260925 03:01:41.739619 6699 catalog_manager.cc:4841] TS dc43ba95d2314b63acc9a8fd20421016 (127.18.196.195:46405): tablet 8850b9ad3de34e23b92816f8179614a8 (table test-table [id=6fcbc8555bf94fe295961665ef677b0b]) successfully deleted
I20260925 03:01:41.825033 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 6772
I20260925 03:01:41.843086 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 6904
W20260925 03:01:41.843138 6698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.196.193:33851 (error 108)
I20260925 03:01:41.844986 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:41.844578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33851 (local address 127.0.0.1:52422)
0925 03:01:41.844739 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:41.844848 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:33851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 03:01:41.845222 6698 catalog_manager.cc:4584] TS 7c27192b09d1418b8e18f1b004bc2059 (127.18.196.193:33851): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8850b9ad3de34e23b92816f8179614a8: Network error: Client connection negotiation failed: client connection to 127.18.196.193:33851: connect: Connection refused (error 111)
I20260925 03:01:41.850132 7177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:41.843890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49444 (local address 127.18.196.194:39697)
0925 03:01:41.844090 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:41.844097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:41.844121 (+ 24us) server_negotiation.cc:407] Connection header received
0925 03:01:41.844282 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:41.844288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:41.844390 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:41.844505 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:41.844912 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.845901 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.847291 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.847707 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.847842 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:41.848818 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:41.848827 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:41.848840 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:41.849394 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:41.849453 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:41.849473 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:41.849592 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:41.849720 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 03:01:41.849824 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 03:01:41.850765 7027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:41.898941 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:41.898354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33851 (local address 127.0.0.1:52432)
0925 03:01:41.898604 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:41.898712 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:33851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260925 03:01:41.900060 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:41.899719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33851 (local address 127.0.0.1:52436)
0925 03:01:41.899867 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:41.899954 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:33851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 03:01:41.938580 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 6904
I20260925 03:01:41.955960 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 7035
I20260925 03:01:41.960651 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:41.960076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33851 (local address 127.0.0.1:52446)
0925 03:01:41.960288 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:41.960392 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:33851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 03:01:41.961617 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:41.961307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33851 (local address 127.0.0.1:52462)
0925 03:01:41.961433 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:41.961502 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:33851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 03:01:41.966257 7176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:41.956690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43376 (local address 127.18.196.195:46405)
0925 03:01:41.956883 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:41.956888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:41.956907 (+ 19us) server_negotiation.cc:407] Connection header received
0925 03:01:41.957015 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:41.957019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:41.957106 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:41.957204 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:41.957567 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.958294 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.959560 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:41.960105 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:41.960237 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:41.965019 (+ 4782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:41.965028 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:41.965042 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:41.965592 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:41.965639 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:41.965658 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:41.965751 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:41.965858 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 03:01:41.965942 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 03:01:41.966872 7157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:42.042058 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:42.041592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33851 (local address 127.0.0.1:52468)
0925 03:01:42.041794 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:42.041911 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:33851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 03:01:42.043401 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:42.042718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.193:33851 (local address 127.0.0.1:52480)
0925 03:01:42.042962 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 03:01:42.043041 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.193:33851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260925 03:01:42.065907 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 7035
I20260925 03:01:42.082998 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 6681
I20260925 03:01:42.084124 6743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:42.192166 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 6681
2026-09-25T03:01:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2702 ms)
[----------] 6 tests from EncryptionPolicyTest (21710 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 03:01:42 dist-test-slave-8zvk krb5kdc[7215](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 03:01:42 dist-test-slave-8zvk krb5kdc[7215](info): set up 2 sockets
Sep 25 03:01:42 dist-test-slave-8zvk krb5kdc[7215](info): commencing operation
krb5kdc: starting...
W20260925 03:01:43.294695 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 03:01:43 dist-test-slave-8zvk krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T03:01:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:01:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 03:01:43.564417 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36999
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:01:43.892958 7228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:43.893227 7228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:43.893334 7228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:43.900847 7228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:01:43.900959 7228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:43.901007 7228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:43.901067 7228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:01:43.901113 7228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:01:43.910465 7228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36999
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:43.911715 7228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:43.912904 7228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:43.924338 7233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:43.924376 7236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:43.924283 7234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:43.925392 7228 server_base.cc:1034] running on GCE node
I20260925 03:01:43.926052 7228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:43.927378 7228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:43.928570 7228 hybrid_clock.cc:648] HybridClock initialized: now 1790305303928531 us; error 39 us; skew 500 ppm
I20260925 03:01:43.928858 7228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:43 dist-test-slave-8zvk krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:43.932281 7228 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 03:01:43.933772 7228 webserver.cc:460] Webserver started at http://127.0.0.1:39011/ using document root <none> and password file <none>
I20260925 03:01:43.934403 7228 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:43.934526 7228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:43.934931 7228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:43.937873 7228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "ec976f2259a34f53b649c863aefff308"
format_stamp: "Formatted at 2026-09-25 03:01:43 on dist-test-slave-8zvk"
server_key: "21e775ca5f275b1fcd28f3ce43ce7fa1"
server_key_iv: "39efa501524661f3acff566eee86e8f2"
server_key_version: "encryptionkey@0"
I20260925 03:01:43.938638 7228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "ec976f2259a34f53b649c863aefff308"
format_stamp: "Formatted at 2026-09-25 03:01:43 on dist-test-slave-8zvk"
server_key: "21e775ca5f275b1fcd28f3ce43ce7fa1"
server_key_iv: "39efa501524661f3acff566eee86e8f2"
server_key_version: "encryptionkey@0"
I20260925 03:01:43.943493 7228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 03:01:43.946672 7243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:43.947948 7228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 03:01:43.948117 7228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "ec976f2259a34f53b649c863aefff308"
format_stamp: "Formatted at 2026-09-25 03:01:43 on dist-test-slave-8zvk"
server_key: "21e775ca5f275b1fcd28f3ce43ce7fa1"
server_key_iv: "39efa501524661f3acff566eee86e8f2"
server_key_version: "encryptionkey@0"
I20260925 03:01:43.948326 7228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:43.972154 7228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:43.976011 7228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:43.976279 7228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:43.993731 7228 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36999
I20260925 03:01:43.993803 7294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36999 every 8 connection(s)
I20260925 03:01:43.995018 7228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:01:43.997460 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 7228
I20260925 03:01:43.997679 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:01:43.998503 19219 external_mini_cluster.cc:1428] Setting key 0bcd5fe0750d7135e702d9e469e4558b
I20260925 03:01:43.999080 7295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:44.013324 7295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec976f2259a34f53b649c863aefff308: Bootstrap starting.
Sep 25 03:01:44 dist-test-slave-8zvk krb5kdc[7215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 03:01:44.017094 7295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec976f2259a34f53b649c863aefff308: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:44.018350 7295 log.cc:824] T 00000000000000000000000000000000 P ec976f2259a34f53b649c863aefff308: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:44.021464 7295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec976f2259a34f53b649c863aefff308: No bootstrap required, opened a new log
I20260925 03:01:44.023183 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:44.000531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37121 (local address 127.0.0.1:36999)
0925 03:01:44.001485 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:44.001501 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:44.001539 (+ 38us) server_negotiation.cc:407] Connection header received
0925 03:01:44.003814 (+ 2275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:44.003835 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:44.004196 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:44.004514 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:44.006210 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:44.007516 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:44.009231 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:44.009886 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:44.010029 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:44.014743 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:44.014756 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:44.014774 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:44.014825 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:44.018645 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:44.019617 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:44.019624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:44.019631 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:44.019823 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:44.020232 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:44.020237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:44.020240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:44.020868 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:44.021151 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:44.021541 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 03:01:44.021877 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":498,"threads_started":1}
I20260925 03:01:44.026880 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 03:01:44.030531 7295 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec976f2259a34f53b649c863aefff308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec976f2259a34f53b649c863aefff308" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36999 } }
I20260925 03:01:44.031015 7295 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec976f2259a34f53b649c863aefff308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:44.031117 7295 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec976f2259a34f53b649c863aefff308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec976f2259a34f53b649c863aefff308, State: Initialized, Role: FOLLOWER
I20260925 03:01:44.031672 7295 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec976f2259a34f53b649c863aefff308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec976f2259a34f53b649c863aefff308" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36999 } }
I20260925 03:01:44.031957 7295 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec976f2259a34f53b649c863aefff308 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:01:44.032105 7295 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec976f2259a34f53b649c863aefff308 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:01:44.032291 7295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec976f2259a34f53b649c863aefff308 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:44.034602 7295 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec976f2259a34f53b649c863aefff308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec976f2259a34f53b649c863aefff308" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36999 } }
I20260925 03:01:44.035157 7295 leader_election.cc:302] T 00000000000000000000000000000000 P ec976f2259a34f53b649c863aefff308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec976f2259a34f53b649c863aefff308; no voters:
I20260925 03:01:44.035756 7295 leader_election.cc:288] T 00000000000000000000000000000000 P ec976f2259a34f53b649c863aefff308 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:01:44.035900 7305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec976f2259a34f53b649c863aefff308 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:01:44.037313 7305 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec976f2259a34f53b649c863aefff308 [term 1 LEADER]: Becoming Leader. State: Replica: ec976f2259a34f53b649c863aefff308, State: Running, Role: LEADER
I20260925 03:01:44.038072 7295 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec976f2259a34f53b649c863aefff308 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:01:44.037981 7305 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec976f2259a34f53b649c863aefff308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec976f2259a34f53b649c863aefff308" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36999 } }
I20260925 03:01:44.047261 7307 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec976f2259a34f53b649c863aefff308 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec976f2259a34f53b649c863aefff308. Latest consensus state: current_term: 1 leader_uuid: "ec976f2259a34f53b649c863aefff308" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec976f2259a34f53b649c863aefff308" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36999 } } }
I20260925 03:01:44.047592 7307 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec976f2259a34f53b649c863aefff308 [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:44.047160 7306 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec976f2259a34f53b649c863aefff308 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec976f2259a34f53b649c863aefff308" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec976f2259a34f53b649c863aefff308" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36999 } } }
I20260925 03:01:44.048054 7306 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec976f2259a34f53b649c863aefff308 [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:44.051998 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:44.032227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49927 (local address 127.0.0.1:36999)
0925 03:01:44.032376 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:44.032383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:44.032869 (+ 486us) server_negotiation.cc:407] Connection header received
0925 03:01:44.032982 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:44.032986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:44.033050 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:44.033151 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:44.034372 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:44.035466 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:44.037537 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:44.038386 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:44.038509 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:44.041985 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:44.041994 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:44.042002 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:44.042048 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:44.046421 (+ 4373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:44.047728 (+ 1307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:44.047737 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:44.047742 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:44.047916 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:44.048657 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:44.048664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:44.048670 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:44.049539 (+ 869us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:44.049863 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:44.051280 (+ 1417us) server_negotiation.cc:299] Negotiation successful
0925 03:01:44.051524 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 03:01:44.058789 7321 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec976f2259a34f53b649c863aefff308: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 03:01:44.058984 7321 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 03:01:44.059150 7316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:01:44.060837 7316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:01:44.068908 7316 catalog_manager.cc:1347] Generated new cluster ID: d37a676bdbbb4ba895a3663ffadeddad
I20260925 03:01:44.069010 7316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:01:44.091713 7316 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:01:44.093046 7316 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:01:44.115584 7316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec976f2259a34f53b649c863aefff308: Generated new TSK 0
I20260925 03:01:44.116460 7316 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:01:44.128918 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 7228
I20260925 03:01:44.130887 7290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:44.223366 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 7228
W20260925 03:01:44.240545 7213 connection.cc:535] client connection to 127.0.0.1:36999 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36999): Connection reset by peer (error 104)
2026-09-25T03:01:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2029 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 03:01:44 dist-test-slave-8zvk krb5kdc[7327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 03:01:44 dist-test-slave-8zvk krb5kdc[7327](info): set up 2 sockets
Sep 25 03:01:44 dist-test-slave-8zvk krb5kdc[7327](info): commencing operation
krb5kdc: starting...
W20260925 03:01:45.326951 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.009s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 03:01:45 dist-test-slave-8zvk krb5kdc[7327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T03:01:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:01:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 03:01:45.578994 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43943
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:01:45.902052 7340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:45.902379 7340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:45.902536 7340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:45.909569 7340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:01:45.909709 7340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:45.909816 7340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:45.909925 7340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:01:45.910009 7340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:01:45.919592 7340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43943
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:45.920869 7340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:45.922122 7340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:45.934098 7345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:45.934258 7346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:45.934240 7348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:45.935205 7340 server_base.cc:1034] running on GCE node
I20260925 03:01:45.935974 7340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:45.937341 7340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:45.938575 7340 hybrid_clock.cc:648] HybridClock initialized: now 1790305305938539 us; error 56 us; skew 500 ppm
I20260925 03:01:45.938975 7340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:45 dist-test-slave-8zvk krb5kdc[7327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:45.942131 7340 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 03:01:45.943643 7340 webserver.cc:460] Webserver started at http://127.0.0.1:43525/ using document root <none> and password file <none>
I20260925 03:01:45.944278 7340 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:45.944428 7340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:45.944720 7340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:45.947666 7340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "cd9ea7fe518a4c178d72c6d82a9cc6d7"
format_stamp: "Formatted at 2026-09-25 03:01:45 on dist-test-slave-8zvk"
server_key: "04452ad7e74c77fc01bc9d748228d35e"
server_key_iv: "02c75286d5bf6d38e3381aec62b8b8ab"
server_key_version: "encryptionkey@0"
I20260925 03:01:45.948421 7340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "cd9ea7fe518a4c178d72c6d82a9cc6d7"
format_stamp: "Formatted at 2026-09-25 03:01:45 on dist-test-slave-8zvk"
server_key: "04452ad7e74c77fc01bc9d748228d35e"
server_key_iv: "02c75286d5bf6d38e3381aec62b8b8ab"
server_key_version: "encryptionkey@0"
I20260925 03:01:45.953256 7340 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 03:01:45.956636 7355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:45.957965 7340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 03:01:45.958158 7340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "cd9ea7fe518a4c178d72c6d82a9cc6d7"
format_stamp: "Formatted at 2026-09-25 03:01:45 on dist-test-slave-8zvk"
server_key: "04452ad7e74c77fc01bc9d748228d35e"
server_key_iv: "02c75286d5bf6d38e3381aec62b8b8ab"
server_key_version: "encryptionkey@0"
I20260925 03:01:45.958357 7340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:45.978048 7340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:45.982142 7340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:45.982411 7340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:46.001212 7340 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43943
I20260925 03:01:46.001251 7406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43943 every 8 connection(s)
I20260925 03:01:46.002547 7340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:01:46.006140 7407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:46.010988 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 7340
I20260925 03:01:46.011181 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:01:46.011745 19219 external_mini_cluster.cc:1428] Setting key 2e6f00fdcd665dd62b96b75ea802f974
I20260925 03:01:46.017506 7407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd9ea7fe518a4c178d72c6d82a9cc6d7: Bootstrap starting.
I20260925 03:01:46.021092 7407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd9ea7fe518a4c178d72c6d82a9cc6d7: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:46.022404 7407 log.cc:824] T 00000000000000000000000000000000 P cd9ea7fe518a4c178d72c6d82a9cc6d7: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:46.025250 7410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:46.013434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45407 (local address 127.0.0.1:43943)
0925 03:01:46.014246 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:46.014256 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:46.014297 (+ 41us) server_negotiation.cc:407] Connection header received
0925 03:01:46.016078 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:46.016122 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:46.016492 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:46.016783 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:46.017373 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:46.018710 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:46.020643 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:46.021351 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:46.021494 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:46.022750 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:46.022758 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:46.022771 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:46.023589 (+ 818us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:46.023695 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:46.023731 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:46.023881 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:46.024150 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 03:01:46.024340 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":388,"threads_started":1}
I20260925 03:01:46.025552 7407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd9ea7fe518a4c178d72c6d82a9cc6d7: No bootstrap required, opened a new log
I20260925 03:01:46.029075 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 03:01:46.033780 7407 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd9ea7fe518a4c178d72c6d82a9cc6d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9ea7fe518a4c178d72c6d82a9cc6d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43943 } }
I20260925 03:01:46.034276 7407 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd9ea7fe518a4c178d72c6d82a9cc6d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:46.034407 7407 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd9ea7fe518a4c178d72c6d82a9cc6d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd9ea7fe518a4c178d72c6d82a9cc6d7, State: Initialized, Role: FOLLOWER
I20260925 03:01:46.035007 7407 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd9ea7fe518a4c178d72c6d82a9cc6d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9ea7fe518a4c178d72c6d82a9cc6d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43943 } }
I20260925 03:01:46.035269 7407 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd9ea7fe518a4c178d72c6d82a9cc6d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:01:46.035429 7407 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd9ea7fe518a4c178d72c6d82a9cc6d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:01:46.035609 7407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd9ea7fe518a4c178d72c6d82a9cc6d7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:46.037735 7407 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd9ea7fe518a4c178d72c6d82a9cc6d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9ea7fe518a4c178d72c6d82a9cc6d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43943 } }
I20260925 03:01:46.038216 7407 leader_election.cc:302] T 00000000000000000000000000000000 P cd9ea7fe518a4c178d72c6d82a9cc6d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd9ea7fe518a4c178d72c6d82a9cc6d7; no voters:
I20260925 03:01:46.038981 7407 leader_election.cc:288] T 00000000000000000000000000000000 P cd9ea7fe518a4c178d72c6d82a9cc6d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:01:46.039064 7417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd9ea7fe518a4c178d72c6d82a9cc6d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:01:46.039760 7417 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd9ea7fe518a4c178d72c6d82a9cc6d7 [term 1 LEADER]: Becoming Leader. State: Replica: cd9ea7fe518a4c178d72c6d82a9cc6d7, State: Running, Role: LEADER
I20260925 03:01:46.039991 7410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:46.033382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43049 (local address 127.0.0.1:43943)
0925 03:01:46.033571 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:46.033578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:46.033916 (+ 338us) server_negotiation.cc:407] Connection header received
0925 03:01:46.034088 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:46.034092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:46.034150 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:46.034256 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:46.034936 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:46.035841 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:46.037126 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:46.037574 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:46.037705 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:46.038883 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:46.038888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:46.038893 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:46.038930 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:46.038968 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:46.038984 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:46.039115 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:46.039510 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 03:01:46.039574 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 03:01:46.040675 7417 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd9ea7fe518a4c178d72c6d82a9cc6d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9ea7fe518a4c178d72c6d82a9cc6d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43943 } }
I20260925 03:01:46.042955 7417 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd9ea7fe518a4c178d72c6d82a9cc6d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd9ea7fe518a4c178d72c6d82a9cc6d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9ea7fe518a4c178d72c6d82a9cc6d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43943 } } }
I20260925 03:01:46.043354 7417 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd9ea7fe518a4c178d72c6d82a9cc6d7 [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:46.043745 7419 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd9ea7fe518a4c178d72c6d82a9cc6d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd9ea7fe518a4c178d72c6d82a9cc6d7. Latest consensus state: current_term: 1 leader_uuid: "cd9ea7fe518a4c178d72c6d82a9cc6d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9ea7fe518a4c178d72c6d82a9cc6d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43943 } } }
I20260925 03:01:46.044080 7419 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd9ea7fe518a4c178d72c6d82a9cc6d7 [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:46.047559 7407 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd9ea7fe518a4c178d72c6d82a9cc6d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:01:46.047765 7421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:01:46.052695 7421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:01:46.061226 7421 catalog_manager.cc:1347] Generated new cluster ID: e240b3c1f2d64260b1ba88a273e2ae2a
I20260925 03:01:46.061347 7421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:01:46.073215 7421 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:01:46.074735 7421 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:01:46.085464 7421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd9ea7fe518a4c178d72c6d82a9cc6d7: Generated new TSK 0
I20260925 03:01:46.086263 7421 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:01:46.099952 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 7340
W20260925 03:01:46.101166 7402 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45407
W20260925 03:01:46.101997 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 7340: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 03:01:46.102169 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 7340
W20260925 03:01:46.119401 7325 connection.cc:535] client connection to 127.0.0.1:43943 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43943): Connection reset by peer (error 104)
2026-09-25T03:01:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1874 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 03:01:46 dist-test-slave-8zvk krb5kdc[7438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 03:01:46 dist-test-slave-8zvk krb5kdc[7438](info): set up 2 sockets
Sep 25 03:01:46 dist-test-slave-8zvk krb5kdc[7438](info): commencing operation
krb5kdc: starting...
W20260925 03:01:47.206499 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 03:01:47 dist-test-slave-8zvk krb5kdc[7438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T03:01:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:01:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 03:01:47.466528 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46235
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:01:47.800390 7451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:47.800684 7451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:47.800776 7451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:47.808048 7451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:01:47.808161 7451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:47.808205 7451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:47.808266 7451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:01:47.808311 7451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:01:47.818257 7451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46235
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:47.819628 7451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:47.820941 7451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:47.832304 7457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:47.832370 7456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:47.832348 7459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:47.833247 7451 server_base.cc:1034] running on GCE node
I20260925 03:01:47.833946 7451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:47.835386 7451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:47.836614 7451 hybrid_clock.cc:648] HybridClock initialized: now 1790305307836575 us; error 27 us; skew 500 ppm
I20260925 03:01:47.836994 7451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:47 dist-test-slave-8zvk krb5kdc[7438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:47.843340 7451 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 03:01:47.844956 7451 webserver.cc:460] Webserver started at http://127.0.0.1:40201/ using document root <none> and password file <none>
I20260925 03:01:47.845642 7451 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:47.845772 7451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:47.846076 7451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:47.849057 7451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "cbeac889c8f74ab6b3ece9c5ed528c4c"
format_stamp: "Formatted at 2026-09-25 03:01:47 on dist-test-slave-8zvk"
server_key: "6ce56eed92fad1831aad679661d6c2e7"
server_key_iv: "94d71363e0e2f25f75cf7c9b96f26210"
server_key_version: "encryptionkey@0"
I20260925 03:01:47.849802 7451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "cbeac889c8f74ab6b3ece9c5ed528c4c"
format_stamp: "Formatted at 2026-09-25 03:01:47 on dist-test-slave-8zvk"
server_key: "6ce56eed92fad1831aad679661d6c2e7"
server_key_iv: "94d71363e0e2f25f75cf7c9b96f26210"
server_key_version: "encryptionkey@0"
I20260925 03:01:47.855062 7451 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 03:01:47.858580 7467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:47.860052 7451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 03:01:47.860246 7451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "cbeac889c8f74ab6b3ece9c5ed528c4c"
format_stamp: "Formatted at 2026-09-25 03:01:47 on dist-test-slave-8zvk"
server_key: "6ce56eed92fad1831aad679661d6c2e7"
server_key_iv: "94d71363e0e2f25f75cf7c9b96f26210"
server_key_version: "encryptionkey@0"
I20260925 03:01:47.860469 7451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:47.866355 7451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:47.870288 7451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:47.870587 7451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:47.889365 7451 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46235
I20260925 03:01:47.889447 7518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46235 every 8 connection(s)
I20260925 03:01:47.890784 7451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:01:47.891211 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 7451
I20260925 03:01:47.891393 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:01:47.892066 19219 external_mini_cluster.cc:1428] Setting key 46cf44c7b8d0fba930874dbc4bfce8cd
I20260925 03:01:47.895394 7519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:47.899762 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:47.893968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41253 (local address 127.0.0.1:46235)
0925 03:01:47.894752 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:47.894765 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:47.894806 (+ 41us) server_negotiation.cc:407] Connection header received
0925 03:01:47.896535 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:47.896558 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:47.896864 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:47.897175 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:47.898201 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:47.898209 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:47.898221 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:47.898240 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:47.898311 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:47.898330 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:47.898495 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:47.898600 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 03:01:47.898791 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":357,"threads_started":1}
I20260925 03:01:47.903553 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 03:01:47.908953 7519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbeac889c8f74ab6b3ece9c5ed528c4c: Bootstrap starting.
I20260925 03:01:47.911052 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:47.908699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36749 (local address 127.0.0.1:46235)
0925 03:01:47.908874 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:47.908879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:47.909222 (+ 343us) server_negotiation.cc:407] Connection header received
0925 03:01:47.909332 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:47.909337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:47.909398 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:47.909523 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:47.910362 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:47.910370 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:47.910376 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:47.910403 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:47.910455 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:47.910484 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:47.910621 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:47.910666 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 03:01:47.910720 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 03:01:47.912737 7519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbeac889c8f74ab6b3ece9c5ed528c4c: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:47.913740 7519 log.cc:824] T 00000000000000000000000000000000 P cbeac889c8f74ab6b3ece9c5ed528c4c: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:47.916374 7519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbeac889c8f74ab6b3ece9c5ed528c4c: No bootstrap required, opened a new log
I20260925 03:01:47.924441 7519 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbeac889c8f74ab6b3ece9c5ed528c4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbeac889c8f74ab6b3ece9c5ed528c4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46235 } }
I20260925 03:01:47.924814 7519 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbeac889c8f74ab6b3ece9c5ed528c4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:47.924942 7519 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbeac889c8f74ab6b3ece9c5ed528c4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbeac889c8f74ab6b3ece9c5ed528c4c, State: Initialized, Role: FOLLOWER
I20260925 03:01:47.925565 7519 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbeac889c8f74ab6b3ece9c5ed528c4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbeac889c8f74ab6b3ece9c5ed528c4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46235 } }
I20260925 03:01:47.925800 7519 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbeac889c8f74ab6b3ece9c5ed528c4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:01:47.925956 7519 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbeac889c8f74ab6b3ece9c5ed528c4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:01:47.926137 7519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbeac889c8f74ab6b3ece9c5ed528c4c [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:47.928406 7519 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbeac889c8f74ab6b3ece9c5ed528c4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbeac889c8f74ab6b3ece9c5ed528c4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46235 } }
I20260925 03:01:47.928907 7519 leader_election.cc:302] T 00000000000000000000000000000000 P cbeac889c8f74ab6b3ece9c5ed528c4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbeac889c8f74ab6b3ece9c5ed528c4c; no voters:
I20260925 03:01:47.929543 7519 leader_election.cc:288] T 00000000000000000000000000000000 P cbeac889c8f74ab6b3ece9c5ed528c4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:01:47.929699 7529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbeac889c8f74ab6b3ece9c5ed528c4c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:01:47.930430 7529 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbeac889c8f74ab6b3ece9c5ed528c4c [term 1 LEADER]: Becoming Leader. State: Replica: cbeac889c8f74ab6b3ece9c5ed528c4c, State: Running, Role: LEADER
I20260925 03:01:47.931119 7529 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbeac889c8f74ab6b3ece9c5ed528c4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbeac889c8f74ab6b3ece9c5ed528c4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46235 } }
I20260925 03:01:47.931807 7519 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbeac889c8f74ab6b3ece9c5ed528c4c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:01:47.933795 7531 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbeac889c8f74ab6b3ece9c5ed528c4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbeac889c8f74ab6b3ece9c5ed528c4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbeac889c8f74ab6b3ece9c5ed528c4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46235 } } }
I20260925 03:01:47.934405 7531 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbeac889c8f74ab6b3ece9c5ed528c4c [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:47.933974 7530 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbeac889c8f74ab6b3ece9c5ed528c4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbeac889c8f74ab6b3ece9c5ed528c4c. Latest consensus state: current_term: 1 leader_uuid: "cbeac889c8f74ab6b3ece9c5ed528c4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbeac889c8f74ab6b3ece9c5ed528c4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46235 } } }
I20260925 03:01:47.934978 7530 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbeac889c8f74ab6b3ece9c5ed528c4c [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:47.939930 7537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:01:47.945253 7537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:01:47.954375 7537 catalog_manager.cc:1347] Generated new cluster ID: 6fd42e33b5ab48e7bf868cdce8c9f5ee
I20260925 03:01:47.954504 7537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:01:47.965909 7537 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:01:47.966323 7537 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:01:47.974229 7537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbeac889c8f74ab6b3ece9c5ed528c4c: Generated new TSK 0
I20260925 03:01:47.975091 7537 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:01:47.983366 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 7451
W20260925 03:01:47.984974 7514 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41253
W20260925 03:01:47.985720 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 7451: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 03:01:47.985874 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 7451
2026-09-25T03:01:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1883 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 03:01:48 dist-test-slave-8zvk krb5kdc[7550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 03:01:48 dist-test-slave-8zvk krb5kdc[7550](info): set up 2 sockets
Sep 25 03:01:48 dist-test-slave-8zvk krb5kdc[7550](info): commencing operation
krb5kdc: starting...
W20260925 03:01:49.082290 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 03:01:49 dist-test-slave-8zvk krb5kdc[7550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T03:01:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:01:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 03:01:49.334091 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41549
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:01:49.659096 7563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:49.659402 7563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:49.659574 7563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:49.666675 7563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:01:49.666817 7563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:49.666934 7563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:49.667042 7563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:01:49.667109 7563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:01:49.676723 7563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:49.678020 7563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:49.679346 7563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:49.691161 7568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:49.691263 7571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:49.691080 7569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:49.692500 7563 server_base.cc:1034] running on GCE node
I20260925 03:01:49.693315 7563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:49.694746 7563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:49.695986 7563 hybrid_clock.cc:648] HybridClock initialized: now 1790305309695943 us; error 37 us; skew 500 ppm
I20260925 03:01:49.696358 7563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:49 dist-test-slave-8zvk krb5kdc[7550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:49.699647 7563 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 03:01:49.701160 7563 webserver.cc:460] Webserver started at http://127.0.0.1:39255/ using document root <none> and password file <none>
I20260925 03:01:49.701858 7563 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:49.702013 7563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:49.702315 7563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:49.705340 7563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "4f743c92c0514d6486a589e77789f696"
format_stamp: "Formatted at 2026-09-25 03:01:49 on dist-test-slave-8zvk"
server_key: "da9e1e2a37cf40afca9a9dc1d99e78d2"
server_key_iv: "d85fa3635786f57a3ea2e3177de0e4ce"
server_key_version: "encryptionkey@0"
I20260925 03:01:49.706135 7563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "4f743c92c0514d6486a589e77789f696"
format_stamp: "Formatted at 2026-09-25 03:01:49 on dist-test-slave-8zvk"
server_key: "da9e1e2a37cf40afca9a9dc1d99e78d2"
server_key_iv: "d85fa3635786f57a3ea2e3177de0e4ce"
server_key_version: "encryptionkey@0"
I20260925 03:01:49.711239 7563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 03:01:49.714583 7578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:49.715973 7563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 03:01:49.716159 7563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "4f743c92c0514d6486a589e77789f696"
format_stamp: "Formatted at 2026-09-25 03:01:49 on dist-test-slave-8zvk"
server_key: "da9e1e2a37cf40afca9a9dc1d99e78d2"
server_key_iv: "d85fa3635786f57a3ea2e3177de0e4ce"
server_key_version: "encryptionkey@0"
I20260925 03:01:49.716372 7563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:49.732831 7563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:49.737089 7563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:49.737377 7563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:49.755329 7563 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41549
I20260925 03:01:49.755359 7629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41549 every 8 connection(s)
I20260925 03:01:49.756639 7563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:01:49.760087 7630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:49.766304 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 7563
I20260925 03:01:49.766511 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:01:49.767113 19219 external_mini_cluster.cc:1428] Setting key f0b434001de56a85e0b0b7ebf3b452f8
I20260925 03:01:49.770648 7630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f743c92c0514d6486a589e77789f696: Bootstrap starting.
I20260925 03:01:49.774281 7630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f743c92c0514d6486a589e77789f696: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:49.775440 7630 log.cc:824] T 00000000000000000000000000000000 P 4f743c92c0514d6486a589e77789f696: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:49.777990 7630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f743c92c0514d6486a589e77789f696: No bootstrap required, opened a new log
Sep 25 03:01:49 dist-test-slave-8zvk krb5kdc[7550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 03:01:49.785187 7630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f743c92c0514d6486a589e77789f696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f743c92c0514d6486a589e77789f696" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41549 } }
I20260925 03:01:49.785542 7630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f743c92c0514d6486a589e77789f696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:49.785643 7630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f743c92c0514d6486a589e77789f696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f743c92c0514d6486a589e77789f696, State: Initialized, Role: FOLLOWER
I20260925 03:01:49.786166 7630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f743c92c0514d6486a589e77789f696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f743c92c0514d6486a589e77789f696" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41549 } }
I20260925 03:01:49.786433 7630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f743c92c0514d6486a589e77789f696 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:01:49.786580 7630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f743c92c0514d6486a589e77789f696 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:01:49.786749 7630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f743c92c0514d6486a589e77789f696 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:49.789158 7630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f743c92c0514d6486a589e77789f696 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f743c92c0514d6486a589e77789f696" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41549 } }
I20260925 03:01:49.789508 7633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:49.768790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57413 (local address 127.0.0.1:41549)
0925 03:01:49.769515 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:49.769526 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:49.769564 (+ 38us) server_negotiation.cc:407] Connection header received
0925 03:01:49.772007 (+ 2443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:49.772035 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:49.772522 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:49.772946 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:49.774499 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:49.775981 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:49.777250 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:49.777903 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:49.778106 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:49.782083 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:49.782091 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:49.782105 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:49.782141 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:49.785281 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:49.786151 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:49.786161 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:49.786166 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:49.786342 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:49.786799 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:49.786805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:49.786814 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:49.787381 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:49.787720 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:49.788119 (+ 399us) server_negotiation.cc:299] Negotiation successful
0925 03:01:49.788536 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":338,"threads_started":1}
I20260925 03:01:49.789690 7630 leader_election.cc:302] T 00000000000000000000000000000000 P 4f743c92c0514d6486a589e77789f696 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f743c92c0514d6486a589e77789f696; no voters:
I20260925 03:01:49.790359 7630 leader_election.cc:288] T 00000000000000000000000000000000 P 4f743c92c0514d6486a589e77789f696 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:01:49.790423 7636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f743c92c0514d6486a589e77789f696 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:01:49.792475 7636 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f743c92c0514d6486a589e77789f696 [term 1 LEADER]: Becoming Leader. State: Replica: 4f743c92c0514d6486a589e77789f696, State: Running, Role: LEADER
I20260925 03:01:49.792748 7630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f743c92c0514d6486a589e77789f696 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:01:49.793068 7636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f743c92c0514d6486a589e77789f696 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f743c92c0514d6486a589e77789f696" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41549 } }
I20260925 03:01:49.794389 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 03:01:49.795714 7638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f743c92c0514d6486a589e77789f696 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f743c92c0514d6486a589e77789f696. Latest consensus state: current_term: 1 leader_uuid: "4f743c92c0514d6486a589e77789f696" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f743c92c0514d6486a589e77789f696" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41549 } } }
I20260925 03:01:49.796089 7638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f743c92c0514d6486a589e77789f696 [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:49.798588 7637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f743c92c0514d6486a589e77789f696 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f743c92c0514d6486a589e77789f696" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f743c92c0514d6486a589e77789f696" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41549 } } }
I20260925 03:01:49.798971 7637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f743c92c0514d6486a589e77789f696 [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:49.814477 7642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:01:49.819696 7642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:01:49.831182 7633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:49.808090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46345 (local address 127.0.0.1:41549)
0925 03:01:49.808279 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:49.808289 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:49.808868 (+ 579us) server_negotiation.cc:407] Connection header received
0925 03:01:49.809124 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:49.809132 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:49.809249 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:49.809371 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:49.811878 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:49.813327 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:49.815169 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:49.816166 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:49.816328 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:49.820742 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:49.820754 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:49.820762 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:49.820819 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:49.825791 (+ 4972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:49.826978 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:49.826988 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:49.826995 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:49.827188 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:49.827744 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:49.827752 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:49.827758 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:49.828349 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:49.828676 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:49.830280 (+ 1604us) server_negotiation.cc:299] Negotiation successful
0925 03:01:49.830592 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 03:01:49.838714 7642 catalog_manager.cc:1347] Generated new cluster ID: 2b24b2a4128b4c82a121612187d30e5f
I20260925 03:01:49.838828 7642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:01:49.854913 7642 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:01:49.856642 7642 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:01:49.865355 7642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f743c92c0514d6486a589e77789f696: Generated new TSK 0
I20260925 03:01:49.866310 7642 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:01:49.873052 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 7563
I20260925 03:01:49.874284 7625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:49.962339 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 7563
2026-09-25T03:01:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1976 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 03:01:50 dist-test-slave-8zvk krb5kdc[7664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 03:01:50 dist-test-slave-8zvk krb5kdc[7664](info): set up 2 sockets
Sep 25 03:01:50 dist-test-slave-8zvk krb5kdc[7664](info): commencing operation
krb5kdc: starting...
W20260925 03:01:51.058320 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 03:01:51 dist-test-slave-8zvk krb5kdc[7664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T03:01:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:01:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 03:01:51.326292 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36827
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:01:51.647553 7677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:51.647869 7677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:51.648042 7677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:51.655452 7677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:01:51.655606 7677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:51.655709 7677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:51.655812 7677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:01:51.655891 7677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:01:51.665568 7677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:51.666896 7677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:51.668161 7677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:51.679466 7682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:51.679494 7683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:51.679548 7685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:51.680362 7677 server_base.cc:1034] running on GCE node
I20260925 03:01:51.681123 7677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:51.682441 7677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:51.683629 7677 hybrid_clock.cc:648] HybridClock initialized: now 1790305311683576 us; error 44 us; skew 500 ppm
I20260925 03:01:51.683946 7677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:51 dist-test-slave-8zvk krb5kdc[7664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:51.686973 7677 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 03:01:51.688302 7677 webserver.cc:460] Webserver started at http://127.0.0.1:35425/ using document root <none> and password file <none>
I20260925 03:01:51.689008 7677 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:51.689131 7677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:51.689432 7677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:51.692351 7677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "19d2f92731ab413cb50b8144a54838ec"
format_stamp: "Formatted at 2026-09-25 03:01:51 on dist-test-slave-8zvk"
server_key: "69f641eea639f9cbb1fb7af8ac8ab7d9"
server_key_iv: "ea400fb2b33262f55ab60e126e981e45"
server_key_version: "encryptionkey@0"
I20260925 03:01:51.693094 7677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "19d2f92731ab413cb50b8144a54838ec"
format_stamp: "Formatted at 2026-09-25 03:01:51 on dist-test-slave-8zvk"
server_key: "69f641eea639f9cbb1fb7af8ac8ab7d9"
server_key_iv: "ea400fb2b33262f55ab60e126e981e45"
server_key_version: "encryptionkey@0"
I20260925 03:01:51.697986 7677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 03:01:51.701382 7692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:51.702611 7677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 03:01:51.702783 7677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "19d2f92731ab413cb50b8144a54838ec"
format_stamp: "Formatted at 2026-09-25 03:01:51 on dist-test-slave-8zvk"
server_key: "69f641eea639f9cbb1fb7af8ac8ab7d9"
server_key_iv: "ea400fb2b33262f55ab60e126e981e45"
server_key_version: "encryptionkey@0"
I20260925 03:01:51.703023 7677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:51.734212 7677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:51.738125 7677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:51.738416 7677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:51.756328 7677 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36827
I20260925 03:01:51.756364 7743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36827 every 8 connection(s)
I20260925 03:01:51.757664 7677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:01:51.760171 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 7677
I20260925 03:01:51.760345 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:01:51.760905 19219 external_mini_cluster.cc:1428] Setting key 43dc6bc48c13d3e19bd150d286a09df3
I20260925 03:01:51.761754 7744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:51.775511 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:51.762597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39183 (local address 127.0.0.1:36827)
0925 03:01:51.763578 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:51.763593 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:51.763884 (+ 291us) server_negotiation.cc:407] Connection header received
0925 03:01:51.766411 (+ 2527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:51.766432 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:51.766803 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:51.767211 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:51.767773 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:51.769068 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:51.771006 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:51.771692 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:51.771845 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:51.773104 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:51.773114 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:51.773127 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:51.773686 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:51.773765 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:51.773795 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:51.773886 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:51.774156 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 03:01:51.774506 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":415,"threads_started":1}
I20260925 03:01:51.775516 7744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d2f92731ab413cb50b8144a54838ec: Bootstrap starting.
I20260925 03:01:51.779322 7744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19d2f92731ab413cb50b8144a54838ec: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:51.780078 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 03:01:51.780589 7744 log.cc:824] T 00000000000000000000000000000000 P 19d2f92731ab413cb50b8144a54838ec: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:51.783923 7744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d2f92731ab413cb50b8144a54838ec: No bootstrap required, opened a new log
I20260925 03:01:51.791566 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:51.784973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55171 (local address 127.0.0.1:36827)
0925 03:01:51.785156 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:51.785163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:51.785641 (+ 478us) server_negotiation.cc:407] Connection header received
0925 03:01:51.785778 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:51.785784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:51.785874 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:51.785984 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:51.786446 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:51.787611 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:51.788867 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:51.789606 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:51.789737 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:51.790758 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:51.790765 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:51.790770 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:51.790791 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:51.790853 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:51.790875 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:51.790958 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:51.791096 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 03:01:51.791203 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 03:01:51.793603 7744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19d2f92731ab413cb50b8144a54838ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d2f92731ab413cb50b8144a54838ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36827 } }
I20260925 03:01:51.793970 7744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19d2f92731ab413cb50b8144a54838ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:51.794081 7744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19d2f92731ab413cb50b8144a54838ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d2f92731ab413cb50b8144a54838ec, State: Initialized, Role: FOLLOWER
I20260925 03:01:51.794651 7744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19d2f92731ab413cb50b8144a54838ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d2f92731ab413cb50b8144a54838ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36827 } }
I20260925 03:01:51.794994 7744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19d2f92731ab413cb50b8144a54838ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:01:51.795174 7744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19d2f92731ab413cb50b8144a54838ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:01:51.795351 7744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19d2f92731ab413cb50b8144a54838ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:51.797456 7744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19d2f92731ab413cb50b8144a54838ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d2f92731ab413cb50b8144a54838ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36827 } }
I20260925 03:01:51.797880 7744 leader_election.cc:302] T 00000000000000000000000000000000 P 19d2f92731ab413cb50b8144a54838ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19d2f92731ab413cb50b8144a54838ec; no voters:
I20260925 03:01:51.798372 7744 leader_election.cc:288] T 00000000000000000000000000000000 P 19d2f92731ab413cb50b8144a54838ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:01:51.798552 7754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19d2f92731ab413cb50b8144a54838ec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:01:51.799328 7754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19d2f92731ab413cb50b8144a54838ec [term 1 LEADER]: Becoming Leader. State: Replica: 19d2f92731ab413cb50b8144a54838ec, State: Running, Role: LEADER
I20260925 03:01:51.799901 7754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19d2f92731ab413cb50b8144a54838ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d2f92731ab413cb50b8144a54838ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36827 } }
I20260925 03:01:51.800515 7744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19d2f92731ab413cb50b8144a54838ec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:01:51.802971 7756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d2f92731ab413cb50b8144a54838ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19d2f92731ab413cb50b8144a54838ec. Latest consensus state: current_term: 1 leader_uuid: "19d2f92731ab413cb50b8144a54838ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d2f92731ab413cb50b8144a54838ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36827 } } }
I20260925 03:01:51.803068 7755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d2f92731ab413cb50b8144a54838ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19d2f92731ab413cb50b8144a54838ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d2f92731ab413cb50b8144a54838ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36827 } } }
I20260925 03:01:51.803464 7756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d2f92731ab413cb50b8144a54838ec [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:51.803491 7755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d2f92731ab413cb50b8144a54838ec [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:51.808079 7759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:01:51.812749 7759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:01:51.820907 7759 catalog_manager.cc:1347] Generated new cluster ID: d7d5b8b47c30415fa8ca45a3e1b5d099
I20260925 03:01:51.821017 7759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:01:51.829357 7759 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:01:51.831136 7759 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:01:51.840481 7759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19d2f92731ab413cb50b8144a54838ec: Generated new TSK 0
I20260925 03:01:51.841435 7759 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:01:51.849732 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 7677
W20260925 03:01:51.850745 7739 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39183
W20260925 03:01:51.851461 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 7677: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 03:01:51.851611 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 7677
2026-09-25T03:01:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1888 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 03:01:51 dist-test-slave-8zvk krb5kdc[7775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 03:01:51 dist-test-slave-8zvk krb5kdc[7775](info): set up 2 sockets
Sep 25 03:01:51 dist-test-slave-8zvk krb5kdc[7775](info): commencing operation
krb5kdc: starting...
W20260925 03:01:52.938553 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 03:01:53 dist-test-slave-8zvk krb5kdc[7775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T03:01:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:01:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 03:01:53.195667 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35325
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:01:53.541885 7788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:53.542160 7788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:53.542336 7788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:53.549418 7788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:01:53.549536 7788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:53.549629 7788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:53.549726 7788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:01:53.549798 7788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:01:53.559438 7788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:53.560612 7788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:53.561921 7788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:53.573607 7796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:53.573567 7794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:53.573640 7793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:53.574332 7788 server_base.cc:1034] running on GCE node
I20260925 03:01:53.575150 7788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:53.576447 7788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:53.577677 7788 hybrid_clock.cc:648] HybridClock initialized: now 1790305313577634 us; error 55 us; skew 500 ppm
I20260925 03:01:53.578016 7788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:53 dist-test-slave-8zvk krb5kdc[7775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:53.581041 7788 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 03:01:53.582352 7788 webserver.cc:460] Webserver started at http://127.0.0.1:32995/ using document root <none> and password file <none>
I20260925 03:01:53.582993 7788 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:53.583093 7788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:53.583345 7788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:53.586089 7788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "b65a1faea851418586474d9b0b5f06d1"
format_stamp: "Formatted at 2026-09-25 03:01:53 on dist-test-slave-8zvk"
server_key: "8cca41f32d4e8b8421304701c0cdfc1d"
server_key_iv: "c3fb3beccf3a8e8cdc923493cf0fce7e"
server_key_version: "encryptionkey@0"
I20260925 03:01:53.586793 7788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "b65a1faea851418586474d9b0b5f06d1"
format_stamp: "Formatted at 2026-09-25 03:01:53 on dist-test-slave-8zvk"
server_key: "8cca41f32d4e8b8421304701c0cdfc1d"
server_key_iv: "c3fb3beccf3a8e8cdc923493cf0fce7e"
server_key_version: "encryptionkey@0"
I20260925 03:01:53.591753 7788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 03:01:53.595094 7803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:53.596451 7788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 03:01:53.596654 7788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "b65a1faea851418586474d9b0b5f06d1"
format_stamp: "Formatted at 2026-09-25 03:01:53 on dist-test-slave-8zvk"
server_key: "8cca41f32d4e8b8421304701c0cdfc1d"
server_key_iv: "c3fb3beccf3a8e8cdc923493cf0fce7e"
server_key_version: "encryptionkey@0"
I20260925 03:01:53.596890 7788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:53.602545 7788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:53.606357 7788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:53.606655 7788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:53.625375 7788 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35325
I20260925 03:01:53.625432 7854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35325 every 8 connection(s)
I20260925 03:01:53.626699 7788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:01:53.630122 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 7788
I20260925 03:01:53.630300 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:01:53.630379 7855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:53.630895 19219 external_mini_cluster.cc:1428] Setting key a6e06bd90764a1ae0b1a6d2beae7d637
I20260925 03:01:53.638584 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:53.632686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40925 (local address 127.0.0.1:35325)
0925 03:01:53.633572 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:53.633584 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:53.633626 (+ 42us) server_negotiation.cc:407] Connection header received
0925 03:01:53.635288 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:53.635309 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:53.635635 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:53.636044 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:53.637088 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:53.637094 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:53.637106 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:53.637125 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:53.637175 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:53.637193 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:53.637445 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:53.637589 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 03:01:53.637754 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":431,"threads_started":1}
I20260925 03:01:53.642052 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 03:01:53.643590 7855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b65a1faea851418586474d9b0b5f06d1: Bootstrap starting.
I20260925 03:01:53.646869 7855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b65a1faea851418586474d9b0b5f06d1: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:53.648212 7855 log.cc:824] T 00000000000000000000000000000000 P b65a1faea851418586474d9b0b5f06d1: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:53.649374 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:53.646644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46687 (local address 127.0.0.1:35325)
0925 03:01:53.646869 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:53.646876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:53.647417 (+ 541us) server_negotiation.cc:407] Connection header received
0925 03:01:53.647570 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:53.647574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:53.647621 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:53.647697 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:53.648714 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:53.648718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:53.648721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:53.648738 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:53.648773 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:53.648787 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:53.648916 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:53.649051 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 03:01:53.649112 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 03:01:53.651232 7855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b65a1faea851418586474d9b0b5f06d1: No bootstrap required, opened a new log
I20260925 03:01:53.658663 7855 raft_consensus.cc:348] T 00000000000000000000000000000000 P b65a1faea851418586474d9b0b5f06d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65a1faea851418586474d9b0b5f06d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35325 } }
I20260925 03:01:53.659075 7855 raft_consensus.cc:374] T 00000000000000000000000000000000 P b65a1faea851418586474d9b0b5f06d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:53.659178 7855 raft_consensus.cc:729] T 00000000000000000000000000000000 P b65a1faea851418586474d9b0b5f06d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65a1faea851418586474d9b0b5f06d1, State: Initialized, Role: FOLLOWER
I20260925 03:01:53.659847 7855 consensus_queue.cc:260] T 00000000000000000000000000000000 P b65a1faea851418586474d9b0b5f06d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65a1faea851418586474d9b0b5f06d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35325 } }
I20260925 03:01:53.660107 7855 raft_consensus.cc:388] T 00000000000000000000000000000000 P b65a1faea851418586474d9b0b5f06d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:01:53.660265 7855 raft_consensus.cc:482] T 00000000000000000000000000000000 P b65a1faea851418586474d9b0b5f06d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:01:53.660426 7855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b65a1faea851418586474d9b0b5f06d1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:53.662544 7855 raft_consensus.cc:504] T 00000000000000000000000000000000 P b65a1faea851418586474d9b0b5f06d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65a1faea851418586474d9b0b5f06d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35325 } }
I20260925 03:01:53.663103 7855 leader_election.cc:302] T 00000000000000000000000000000000 P b65a1faea851418586474d9b0b5f06d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b65a1faea851418586474d9b0b5f06d1; no voters:
I20260925 03:01:53.663645 7855 leader_election.cc:288] T 00000000000000000000000000000000 P b65a1faea851418586474d9b0b5f06d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:01:53.663787 7865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b65a1faea851418586474d9b0b5f06d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:01:53.664464 7865 raft_consensus.cc:686] T 00000000000000000000000000000000 P b65a1faea851418586474d9b0b5f06d1 [term 1 LEADER]: Becoming Leader. State: Replica: b65a1faea851418586474d9b0b5f06d1, State: Running, Role: LEADER
I20260925 03:01:53.665051 7865 consensus_queue.cc:237] T 00000000000000000000000000000000 P b65a1faea851418586474d9b0b5f06d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65a1faea851418586474d9b0b5f06d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35325 } }
I20260925 03:01:53.665683 7855 sys_catalog.cc:564] T 00000000000000000000000000000000 P b65a1faea851418586474d9b0b5f06d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:01:53.667938 7867 sys_catalog.cc:455] T 00000000000000000000000000000000 P b65a1faea851418586474d9b0b5f06d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b65a1faea851418586474d9b0b5f06d1. Latest consensus state: current_term: 1 leader_uuid: "b65a1faea851418586474d9b0b5f06d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65a1faea851418586474d9b0b5f06d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35325 } } }
I20260925 03:01:53.668412 7867 sys_catalog.cc:458] T 00000000000000000000000000000000 P b65a1faea851418586474d9b0b5f06d1 [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:53.668063 7866 sys_catalog.cc:455] T 00000000000000000000000000000000 P b65a1faea851418586474d9b0b5f06d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b65a1faea851418586474d9b0b5f06d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65a1faea851418586474d9b0b5f06d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35325 } } }
I20260925 03:01:53.668740 7866 sys_catalog.cc:458] T 00000000000000000000000000000000 P b65a1faea851418586474d9b0b5f06d1 [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:53.670014 7874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:01:53.674772 7874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:01:53.683882 7874 catalog_manager.cc:1347] Generated new cluster ID: bebcb71865574f30905c1e0c714757e8
I20260925 03:01:53.684036 7874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:01:53.723295 7874 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:01:53.723723 7874 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:01:53.730631 7874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b65a1faea851418586474d9b0b5f06d1: Generated new TSK 0
I20260925 03:01:53.731626 7874 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:01:53.747751 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 7788
W20260925 03:01:53.748733 7850 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40925
W20260925 03:01:53.749411 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 7788: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 03:01:53.749557 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 7788
2026-09-25T03:01:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1896 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 03:01:53 dist-test-slave-8zvk krb5kdc[7886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 03:01:53 dist-test-slave-8zvk krb5kdc[7886](info): set up 2 sockets
Sep 25 03:01:53 dist-test-slave-8zvk krb5kdc[7886](info): commencing operation
krb5kdc: starting...
W20260925 03:01:54.846588 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 03:01:55 dist-test-slave-8zvk krb5kdc[7886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T03:01:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:01:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 03:01:55.109810 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45235
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:01:55.435972 7899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:55.436220 7899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:55.436416 7899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:55.443400 7899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:01:55.443522 7899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:55.443606 7899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:55.443718 7899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:01:55.443804 7899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:01:55.453065 7899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:55.454200 7899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:55.455472 7899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:55.466674 7904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:55.466712 7907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:55.466898 7905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:55.467501 7899 server_base.cc:1034] running on GCE node
I20260925 03:01:55.468142 7899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:55.469344 7899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:55.470556 7899 hybrid_clock.cc:648] HybridClock initialized: now 1790305315470512 us; error 31 us; skew 500 ppm
I20260925 03:01:55.470892 7899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:55 dist-test-slave-8zvk krb5kdc[7886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:55.473835 7899 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 03:01:55.475188 7899 webserver.cc:460] Webserver started at http://127.0.0.1:40739/ using document root <none> and password file <none>
I20260925 03:01:55.475800 7899 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:55.475947 7899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:55.476238 7899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:55.479065 7899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "708156e0bcb24255b0bfb4a3f6981494"
format_stamp: "Formatted at 2026-09-25 03:01:55 on dist-test-slave-8zvk"
server_key: "a2badc86554e985e9c8e7ec5341f2253"
server_key_iv: "56a96113780e3638ce0d8e72ddc6b99c"
server_key_version: "encryptionkey@0"
I20260925 03:01:55.479816 7899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "708156e0bcb24255b0bfb4a3f6981494"
format_stamp: "Formatted at 2026-09-25 03:01:55 on dist-test-slave-8zvk"
server_key: "a2badc86554e985e9c8e7ec5341f2253"
server_key_iv: "56a96113780e3638ce0d8e72ddc6b99c"
server_key_version: "encryptionkey@0"
I20260925 03:01:55.484579 7899 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 03:01:55.487883 7914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:55.489090 7899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 03:01:55.489269 7899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "708156e0bcb24255b0bfb4a3f6981494"
format_stamp: "Formatted at 2026-09-25 03:01:55 on dist-test-slave-8zvk"
server_key: "a2badc86554e985e9c8e7ec5341f2253"
server_key_iv: "56a96113780e3638ce0d8e72ddc6b99c"
server_key_version: "encryptionkey@0"
I20260925 03:01:55.489459 7899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:55.507954 7899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:55.511723 7899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:55.512004 7899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:55.529819 7899 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45235
I20260925 03:01:55.529857 7965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45235 every 8 connection(s)
I20260925 03:01:55.531198 7899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:01:55.533409 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 7899
I20260925 03:01:55.533617 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:01:55.534070 19219 external_mini_cluster.cc:1428] Setting key 8890f6ac7f64b274b6a454ef1e350879
I20260925 03:01:55.535050 7966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 03:01:55 dist-test-slave-8zvk krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 03:01:55.548784 7966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 708156e0bcb24255b0bfb4a3f6981494: Bootstrap starting.
I20260925 03:01:55.552063 7966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 708156e0bcb24255b0bfb4a3f6981494: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:55.553103 7966 log.cc:824] T 00000000000000000000000000000000 P 708156e0bcb24255b0bfb4a3f6981494: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:55.556401 7966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 708156e0bcb24255b0bfb4a3f6981494: No bootstrap required, opened a new log
I20260925 03:01:55.557363 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:55.535903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:39397 (local address 127.0.0.1:45235)
0925 03:01:55.536661 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:55.536676 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:55.536724 (+ 48us) server_negotiation.cc:407] Connection header received
0925 03:01:55.538535 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:55.538562 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:55.539098 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:55.539510 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:55.540845 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:55.542984 (+ 2139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:55.544175 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:55.545188 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:55.545380 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:55.549869 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:55.549877 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:55.549894 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:55.549931 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:55.553056 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:55.554031 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:55.554040 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:55.554049 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:55.554269 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:55.554782 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:55.554785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:55.554787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:55.555304 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:55.555540 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:55.556035 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 03:01:55.556310 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":296,"threads_started":1}
I20260925 03:01:55.561124 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 03:01:55.565871 7966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 708156e0bcb24255b0bfb4a3f6981494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708156e0bcb24255b0bfb4a3f6981494" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45235 } }
I20260925 03:01:55.566308 7966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 708156e0bcb24255b0bfb4a3f6981494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:55.566407 7966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 708156e0bcb24255b0bfb4a3f6981494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 708156e0bcb24255b0bfb4a3f6981494, State: Initialized, Role: FOLLOWER
I20260925 03:01:55.567059 7966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 708156e0bcb24255b0bfb4a3f6981494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708156e0bcb24255b0bfb4a3f6981494" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45235 } }
I20260925 03:01:55.567301 7966 raft_consensus.cc:388] T 00000000000000000000000000000000 P 708156e0bcb24255b0bfb4a3f6981494 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:01:55.567468 7966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 708156e0bcb24255b0bfb4a3f6981494 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:01:55.567651 7966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 708156e0bcb24255b0bfb4a3f6981494 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:55.569765 7966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 708156e0bcb24255b0bfb4a3f6981494 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708156e0bcb24255b0bfb4a3f6981494" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45235 } }
I20260925 03:01:55.570250 7966 leader_election.cc:302] T 00000000000000000000000000000000 P 708156e0bcb24255b0bfb4a3f6981494 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 708156e0bcb24255b0bfb4a3f6981494; no voters:
I20260925 03:01:55.570940 7966 leader_election.cc:288] T 00000000000000000000000000000000 P 708156e0bcb24255b0bfb4a3f6981494 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:01:55.571045 7976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 708156e0bcb24255b0bfb4a3f6981494 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:01:55.573531 7966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 708156e0bcb24255b0bfb4a3f6981494 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:01:55.577721 7976 raft_consensus.cc:686] T 00000000000000000000000000000000 P 708156e0bcb24255b0bfb4a3f6981494 [term 1 LEADER]: Becoming Leader. State: Replica: 708156e0bcb24255b0bfb4a3f6981494, State: Running, Role: LEADER
I20260925 03:01:55.578471 7976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 708156e0bcb24255b0bfb4a3f6981494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708156e0bcb24255b0bfb4a3f6981494" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45235 } }
I20260925 03:01:55.580892 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:55.566243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:57289 (local address 127.0.0.1:45235)
0925 03:01:55.566451 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:55.566457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:55.567124 (+ 667us) server_negotiation.cc:407] Connection header received
0925 03:01:55.567309 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:55.567315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:55.567394 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:55.567508 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:55.568867 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:55.569636 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:55.571013 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:55.571822 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:55.571936 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:55.574669 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:55.574675 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:55.574686 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:01:55.574720 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:55.577769 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:55.578511 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:55.578516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:55.578520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:55.578700 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:01:55.579273 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:01:55.579281 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:01:55.579285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:01:55.579657 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:55.579992 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:55.580354 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 03:01:55.580526 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 03:01:55.589452 7978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 708156e0bcb24255b0bfb4a3f6981494 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 708156e0bcb24255b0bfb4a3f6981494. Latest consensus state: current_term: 1 leader_uuid: "708156e0bcb24255b0bfb4a3f6981494" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708156e0bcb24255b0bfb4a3f6981494" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45235 } } }
I20260925 03:01:55.589846 7978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 708156e0bcb24255b0bfb4a3f6981494 [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:55.589824 7977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 708156e0bcb24255b0bfb4a3f6981494 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "708156e0bcb24255b0bfb4a3f6981494" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708156e0bcb24255b0bfb4a3f6981494" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45235 } } }
I20260925 03:01:55.590152 7977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 708156e0bcb24255b0bfb4a3f6981494 [sys.catalog]: This master's current role is: LEADER
W20260925 03:01:55.596642 7992 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 708156e0bcb24255b0bfb4a3f6981494: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 03:01:55.596781 7992 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 03:01:55.596944 7993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:01:55.599078 7993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:01:55.607410 7993 catalog_manager.cc:1347] Generated new cluster ID: a984ba62ed234eca99da73c79204026d
I20260925 03:01:55.607513 7993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:01:55.623940 7993 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:01:55.625133 7993 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:01:55.634253 7993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 708156e0bcb24255b0bfb4a3f6981494: Generated new TSK 0
I20260925 03:01:55.635082 7993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:01:55.643028 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 7899
I20260925 03:01:55.644220 7961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:01:55.735616 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 7899
W20260925 03:01:55.752918 7884 connection.cc:535] client connection to 127.0.0.1:45235 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45235): Connection reset by peer (error 104)
2026-09-25T03:01:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1988 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 03:01:55 dist-test-slave-8zvk krb5kdc[7999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 03:01:55 dist-test-slave-8zvk krb5kdc[7999](info): set up 2 sockets
Sep 25 03:01:55 dist-test-slave-8zvk krb5kdc[7999](info): commencing operation
krb5kdc: starting...
W20260925 03:01:56.846668 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 03:01:57 dist-test-slave-8zvk krb5kdc[7999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T03:01:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:01:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 03:01:57.102320 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43837
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:01:57.424414 8012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:57.424734 8012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:57.424908 8012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:57.431941 8012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:01:57.432086 8012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:57.432204 8012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:57.432324 8012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:01:57.432415 8012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:01:57.441936 8012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:57.443202 8012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:57.444485 8012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:57.455353 8017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:57.455490 8020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:57.455515 8018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:57.456828 8012 server_base.cc:1034] running on GCE node
I20260925 03:01:57.457614 8012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:57.458920 8012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:57.460125 8012 hybrid_clock.cc:648] HybridClock initialized: now 1790305317460081 us; error 40 us; skew 500 ppm
I20260925 03:01:57.460394 8012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:57 dist-test-slave-8zvk krb5kdc[7999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:57.463276 8012 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 03:01:57.464663 8012 webserver.cc:460] Webserver started at http://127.0.0.1:37709/ using document root <none> and password file <none>
I20260925 03:01:57.465268 8012 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:57.465404 8012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:57.465721 8012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:57.468566 8012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "6c9957c03a154ea08ea33f8ee961d580"
format_stamp: "Formatted at 2026-09-25 03:01:57 on dist-test-slave-8zvk"
server_key: "b6790252aa6d0534aa9e84f6a21004a8"
server_key_iv: "9fb23e69d23beb40f03b698a8d23b6c3"
server_key_version: "encryptionkey@0"
I20260925 03:01:57.469256 8012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "6c9957c03a154ea08ea33f8ee961d580"
format_stamp: "Formatted at 2026-09-25 03:01:57 on dist-test-slave-8zvk"
server_key: "b6790252aa6d0534aa9e84f6a21004a8"
server_key_iv: "9fb23e69d23beb40f03b698a8d23b6c3"
server_key_version: "encryptionkey@0"
I20260925 03:01:57.473740 8012 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 03:01:57.476840 8027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:57.478073 8012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260925 03:01:57.478246 8012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "6c9957c03a154ea08ea33f8ee961d580"
format_stamp: "Formatted at 2026-09-25 03:01:57 on dist-test-slave-8zvk"
server_key: "b6790252aa6d0534aa9e84f6a21004a8"
server_key_iv: "9fb23e69d23beb40f03b698a8d23b6c3"
server_key_version: "encryptionkey@0"
I20260925 03:01:57.478433 8012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:57.494791 8012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:57.498739 8012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:57.499055 8012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:57.515997 8012 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43837
I20260925 03:01:57.516037 8078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43837 every 8 connection(s)
I20260925 03:01:57.517252 8012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:01:57.520691 8079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:57.526589 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 8012
I20260925 03:01:57.526763 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:01:57.527408 19219 external_mini_cluster.cc:1428] Setting key 9c53287880472f1e80b4aedc883a2e82
I20260925 03:01:57.530967 8079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c9957c03a154ea08ea33f8ee961d580: Bootstrap starting.
I20260925 03:01:57.534114 8079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c9957c03a154ea08ea33f8ee961d580: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:57.535377 8079 log.cc:824] T 00000000000000000000000000000000 P 6c9957c03a154ea08ea33f8ee961d580: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:57.538537 8079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c9957c03a154ea08ea33f8ee961d580: No bootstrap required, opened a new log
I20260925 03:01:57.540598 8082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:57.529112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:48705 (local address 127.0.0.1:43837)
0925 03:01:57.529805 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:57.529816 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:57.529853 (+ 37us) server_negotiation.cc:407] Connection header received
0925 03:01:57.531541 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:57.531562 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:57.531902 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:57.532191 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:57.532783 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:57.533974 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:57.535952 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:57.536936 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:57.537079 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:57.538271 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:57.538279 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:57.538291 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:57.538899 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:57.538990 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:57.539035 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:57.539163 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:57.539466 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 03:01:57.539658 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":322,"threads_started":1}
I20260925 03:01:57.544270 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 03:01:57.547487 8079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c9957c03a154ea08ea33f8ee961d580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9957c03a154ea08ea33f8ee961d580" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43837 } }
I20260925 03:01:57.547950 8079 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c9957c03a154ea08ea33f8ee961d580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:57.548094 8079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c9957c03a154ea08ea33f8ee961d580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9957c03a154ea08ea33f8ee961d580, State: Initialized, Role: FOLLOWER
I20260925 03:01:57.548717 8079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c9957c03a154ea08ea33f8ee961d580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9957c03a154ea08ea33f8ee961d580" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43837 } }
I20260925 03:01:57.548986 8079 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c9957c03a154ea08ea33f8ee961d580 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:01:57.549108 8079 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c9957c03a154ea08ea33f8ee961d580 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:01:57.549288 8079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c9957c03a154ea08ea33f8ee961d580 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:57.552208 8079 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c9957c03a154ea08ea33f8ee961d580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9957c03a154ea08ea33f8ee961d580" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43837 } }
I20260925 03:01:57.552771 8079 leader_election.cc:302] T 00000000000000000000000000000000 P 6c9957c03a154ea08ea33f8ee961d580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c9957c03a154ea08ea33f8ee961d580; no voters:
I20260925 03:01:57.553373 8079 leader_election.cc:288] T 00000000000000000000000000000000 P 6c9957c03a154ea08ea33f8ee961d580 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:01:57.553448 8089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c9957c03a154ea08ea33f8ee961d580 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:01:57.555495 8089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c9957c03a154ea08ea33f8ee961d580 [term 1 LEADER]: Becoming Leader. State: Replica: 6c9957c03a154ea08ea33f8ee961d580, State: Running, Role: LEADER
I20260925 03:01:57.556272 8089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c9957c03a154ea08ea33f8ee961d580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9957c03a154ea08ea33f8ee961d580" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43837 } }
I20260925 03:01:57.556452 8079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c9957c03a154ea08ea33f8ee961d580 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:01:57.557039 8082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:57.549482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:34135 (local address 127.0.0.1:43837)
0925 03:01:57.549669 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:57.549678 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:57.550127 (+ 449us) server_negotiation.cc:407] Connection header received
0925 03:01:57.550319 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:57.550323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:57.550387 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:57.550464 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:57.551106 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:57.551849 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:57.553695 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:01:57.554766 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:01:57.554950 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:01:57.556107 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:57.556112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:57.556115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:57.556133 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:57.556206 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:57.556231 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:57.556353 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:57.556673 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 03:01:57.556725 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 03:01:57.559710 8091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c9957c03a154ea08ea33f8ee961d580 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c9957c03a154ea08ea33f8ee961d580. Latest consensus state: current_term: 1 leader_uuid: "6c9957c03a154ea08ea33f8ee961d580" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9957c03a154ea08ea33f8ee961d580" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43837 } } }
I20260925 03:01:57.559993 8090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c9957c03a154ea08ea33f8ee961d580 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c9957c03a154ea08ea33f8ee961d580" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9957c03a154ea08ea33f8ee961d580" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43837 } } }
I20260925 03:01:57.560273 8091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c9957c03a154ea08ea33f8ee961d580 [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:57.560319 8090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c9957c03a154ea08ea33f8ee961d580 [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:57.563028 8095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:01:57.570092 8095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:01:57.578150 8095 catalog_manager.cc:1347] Generated new cluster ID: eae16847bf4d4c498748327a15058ff5
I20260925 03:01:57.578262 8095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:01:57.592830 8095 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:01:57.594496 8095 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:01:57.604544 8095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c9957c03a154ea08ea33f8ee961d580: Generated new TSK 0
I20260925 03:01:57.605517 8095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:01:57.613408 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 8012
W20260925 03:01:57.614423 8074 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.55:48705
W20260925 03:01:57.615227 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 8012: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 03:01:57.615403 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 8012
W20260925 03:01:57.632777 7997 connection.cc:535] client connection to 127.0.0.1:43837 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43837): Connection reset by peer (error 104)
2026-09-25T03:01:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1879 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 03:01:57 dist-test-slave-8zvk krb5kdc[8110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 03:01:57 dist-test-slave-8zvk krb5kdc[8110](info): set up 2 sockets
Sep 25 03:01:57 dist-test-slave-8zvk krb5kdc[8110](info): commencing operation
krb5kdc: starting...
W20260925 03:01:58.690203 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 03:01:58 dist-test-slave-8zvk krb5kdc[8110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T03:01:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:01:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 03:01:58.920272 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39405
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:01:59.234956 8123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:01:59.235260 8123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:01:59.235435 8123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:01:59.242395 8123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:01:59.242489 8123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:01:59.242599 8123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:01:59.242707 8123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:01:59.242784 8123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:01:59.252321 8123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:01:59.253532 8123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:01:59.254794 8123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:01:59.267006 8128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:59.266881 8129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:01:59.267040 8131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:01:59.267743 8123 server_base.cc:1034] running on GCE node
I20260925 03:01:59.268519 8123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:01:59.270076 8123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:01:59.271306 8123 hybrid_clock.cc:648] HybridClock initialized: now 1790305319271258 us; error 58 us; skew 500 ppm
I20260925 03:01:59.271646 8123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:01:59 dist-test-slave-8zvk krb5kdc[8110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:01:59.283540 8123 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 03:01:59.285121 8123 webserver.cc:460] Webserver started at http://127.0.0.1:36699/ using document root <none> and password file <none>
I20260925 03:01:59.285815 8123 fs_manager.cc:359] Metadata directory not provided
I20260925 03:01:59.285970 8123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:01:59.286289 8123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:01:59.289369 8123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "d4b9549f9dff4e55a886f3cf161b8091"
format_stamp: "Formatted at 2026-09-25 03:01:59 on dist-test-slave-8zvk"
server_key: "7a1afa6a7dc45145760af1c38f53adda"
server_key_iv: "1b72107f67f1954418ad71678ce40207"
server_key_version: "encryptionkey@0"
I20260925 03:01:59.290177 8123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "d4b9549f9dff4e55a886f3cf161b8091"
format_stamp: "Formatted at 2026-09-25 03:01:59 on dist-test-slave-8zvk"
server_key: "7a1afa6a7dc45145760af1c38f53adda"
server_key_iv: "1b72107f67f1954418ad71678ce40207"
server_key_version: "encryptionkey@0"
I20260925 03:01:59.295301 8123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 03:01:59.298765 8138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:01:59.300055 8123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 03:01:59.300244 8123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "d4b9549f9dff4e55a886f3cf161b8091"
format_stamp: "Formatted at 2026-09-25 03:01:59 on dist-test-slave-8zvk"
server_key: "7a1afa6a7dc45145760af1c38f53adda"
server_key_iv: "1b72107f67f1954418ad71678ce40207"
server_key_version: "encryptionkey@0"
I20260925 03:01:59.300453 8123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:01:59.305964 8123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:01:59.309826 8123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:01:59.310128 8123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:01:59.328676 8123 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39405
I20260925 03:01:59.328758 8189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39405 every 8 connection(s)
I20260925 03:01:59.330041 8123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:01:59.331360 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 8123
I20260925 03:01:59.331547 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:01:59.332130 19219 external_mini_cluster.cc:1428] Setting key 5030d04057ee7b6f5c20dbe9a57987f0
I20260925 03:01:59.334578 8190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:01:59.339620 8193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:59.333892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:48065 (local address 127.0.0.1:39405)
0925 03:01:59.334698 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:59.334712 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:59.334762 (+ 50us) server_negotiation.cc:407] Connection header received
0925 03:01:59.336380 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:59.336399 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:59.336695 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:59.337010 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:59.338044 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:59.338050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:59.338061 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:59.338079 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:59.338152 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:59.338171 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:59.338325 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:59.338491 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 03:01:59.338718 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":340,"threads_started":1}
I20260925 03:01:59.342983 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 03:01:59.347208 8190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b9549f9dff4e55a886f3cf161b8091: Bootstrap starting.
I20260925 03:01:59.350768 8193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:01:59.348298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:34333 (local address 127.0.0.1:39405)
0925 03:01:59.348515 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 03:01:59.348521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:01:59.348871 (+ 350us) server_negotiation.cc:407] Connection header received
0925 03:01:59.349055 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:01:59.349058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:01:59.349101 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:01:59.349181 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:01:59.350116 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:01:59.350120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:01:59.350123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:01:59.350137 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:01:59.350174 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:01:59.350188 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:01:59.350272 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 03:01:59.350430 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 03:01:59.350476 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 03:01:59.351241 8190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4b9549f9dff4e55a886f3cf161b8091: Neither blocks nor log segments found. Creating new log.
I20260925 03:01:59.352272 8190 log.cc:824] T 00000000000000000000000000000000 P d4b9549f9dff4e55a886f3cf161b8091: Log is configured to *not* fsync() on all Append() calls
I20260925 03:01:59.354823 8190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b9549f9dff4e55a886f3cf161b8091: No bootstrap required, opened a new log
I20260925 03:01:59.361912 8190 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4b9549f9dff4e55a886f3cf161b8091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b9549f9dff4e55a886f3cf161b8091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39405 } }
I20260925 03:01:59.362282 8190 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4b9549f9dff4e55a886f3cf161b8091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:01:59.362391 8190 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4b9549f9dff4e55a886f3cf161b8091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b9549f9dff4e55a886f3cf161b8091, State: Initialized, Role: FOLLOWER
I20260925 03:01:59.362982 8190 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4b9549f9dff4e55a886f3cf161b8091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b9549f9dff4e55a886f3cf161b8091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39405 } }
I20260925 03:01:59.363222 8190 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4b9549f9dff4e55a886f3cf161b8091 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:01:59.363373 8190 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4b9549f9dff4e55a886f3cf161b8091 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:01:59.363538 8190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4b9549f9dff4e55a886f3cf161b8091 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:01:59.365650 8190 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4b9549f9dff4e55a886f3cf161b8091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b9549f9dff4e55a886f3cf161b8091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39405 } }
I20260925 03:01:59.366127 8190 leader_election.cc:302] T 00000000000000000000000000000000 P d4b9549f9dff4e55a886f3cf161b8091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4b9549f9dff4e55a886f3cf161b8091; no voters:
I20260925 03:01:59.366677 8190 leader_election.cc:288] T 00000000000000000000000000000000 P d4b9549f9dff4e55a886f3cf161b8091 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:01:59.366799 8200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4b9549f9dff4e55a886f3cf161b8091 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:01:59.367537 8200 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4b9549f9dff4e55a886f3cf161b8091 [term 1 LEADER]: Becoming Leader. State: Replica: d4b9549f9dff4e55a886f3cf161b8091, State: Running, Role: LEADER
I20260925 03:01:59.368041 8200 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4b9549f9dff4e55a886f3cf161b8091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b9549f9dff4e55a886f3cf161b8091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39405 } }
I20260925 03:01:59.368765 8190 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4b9549f9dff4e55a886f3cf161b8091 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:01:59.370746 8202 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b9549f9dff4e55a886f3cf161b8091 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4b9549f9dff4e55a886f3cf161b8091. Latest consensus state: current_term: 1 leader_uuid: "d4b9549f9dff4e55a886f3cf161b8091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b9549f9dff4e55a886f3cf161b8091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39405 } } }
I20260925 03:01:59.371181 8202 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b9549f9dff4e55a886f3cf161b8091 [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:59.372411 8201 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b9549f9dff4e55a886f3cf161b8091 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4b9549f9dff4e55a886f3cf161b8091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b9549f9dff4e55a886f3cf161b8091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39405 } } }
I20260925 03:01:59.373519 8201 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b9549f9dff4e55a886f3cf161b8091 [sys.catalog]: This master's current role is: LEADER
I20260925 03:01:59.374536 8208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:01:59.379302 8208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:01:59.388026 8208 catalog_manager.cc:1347] Generated new cluster ID: 826c9d4494394bb28022090a4cba4265
I20260925 03:01:59.388135 8208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:01:59.421060 8208 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:01:59.421591 8208 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:01:59.430012 8208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4b9549f9dff4e55a886f3cf161b8091: Generated new TSK 0
I20260925 03:01:59.430900 8208 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:01:59.447045 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 8123
W20260925 03:01:59.448067 8185 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.55:48065
W20260925 03:01:59.448803 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 8123: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 03:01:59.448954 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 8123
2026-09-25T03:01:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1834 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 03:01:59 dist-test-slave-8zvk krb5kdc[8221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 03:01:59 dist-test-slave-8zvk krb5kdc[8221](info): set up 2 sockets
Sep 25 03:01:59 dist-test-slave-8zvk krb5kdc[8221](info): commencing operation
krb5kdc: starting...
W20260925 03:02:00.534140 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 03:02:00 dist-test-slave-8zvk krb5kdc[8221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T03:02:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:02:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 03:02:00.774394 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46595
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:02:01.100721 8234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:02:01.100980 8234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:02:01.101070 8234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:02:01.108227 8234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:02:01.108335 8234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:02:01.108443 8234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:02:01.108580 8234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:02:01.108659 8234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:02:01.118402 8234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46595
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:02:01.119690 8234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:02:01.120918 8234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:02:01.132094 8240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:01.132191 8239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:01.132164 8242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:02:01.133206 8234 server_base.cc:1034] running on GCE node
I20260925 03:02:01.133949 8234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:02:01.135203 8234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:02:01.136406 8234 hybrid_clock.cc:648] HybridClock initialized: now 1790305321136370 us; error 23 us; skew 500 ppm
I20260925 03:02:01.136783 8234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:02:01 dist-test-slave-8zvk krb5kdc[8221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:02:01.139755 8234 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 03:02:01.141108 8234 webserver.cc:460] Webserver started at http://127.0.0.1:33623/ using document root <none> and password file <none>
I20260925 03:02:01.141743 8234 fs_manager.cc:359] Metadata directory not provided
I20260925 03:02:01.141897 8234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:02:01.142198 8234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:02:01.145088 8234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "a07d98f0d1ef45fd8d262c9138c89e30"
format_stamp: "Formatted at 2026-09-25 03:02:01 on dist-test-slave-8zvk"
server_key: "ec5df1b315828f6821def028311f5036"
server_key_iv: "bddca148b95e5f5b7fe706468e51db08"
server_key_version: "encryptionkey@0"
I20260925 03:02:01.145848 8234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "a07d98f0d1ef45fd8d262c9138c89e30"
format_stamp: "Formatted at 2026-09-25 03:02:01 on dist-test-slave-8zvk"
server_key: "ec5df1b315828f6821def028311f5036"
server_key_iv: "bddca148b95e5f5b7fe706468e51db08"
server_key_version: "encryptionkey@0"
I20260925 03:02:01.150660 8234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260925 03:02:01.153968 8249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:01.155186 8234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 03:02:01.155355 8234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "a07d98f0d1ef45fd8d262c9138c89e30"
format_stamp: "Formatted at 2026-09-25 03:02:01 on dist-test-slave-8zvk"
server_key: "ec5df1b315828f6821def028311f5036"
server_key_iv: "bddca148b95e5f5b7fe706468e51db08"
server_key_version: "encryptionkey@0"
I20260925 03:02:01.155540 8234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:02:01.167629 8234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:02:01.171233 8234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:02:01.171494 8234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:02:01.189929 8234 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46595
I20260925 03:02:01.189968 8301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46595 every 8 connection(s)
I20260925 03:02:01.191239 8234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:02:01.194670 8302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:01.199947 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 8234
I20260925 03:02:01.200134 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:02:01.200549 19219 external_mini_cluster.cc:1428] Setting key c677db993fa8a5420bf4da021b357a1c
I20260925 03:02:01.205176 8302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07d98f0d1ef45fd8d262c9138c89e30: Bootstrap starting.
I20260925 03:02:01.208753 8302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a07d98f0d1ef45fd8d262c9138c89e30: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:01.209955 8302 log.cc:824] T 00000000000000000000000000000000 P a07d98f0d1ef45fd8d262c9138c89e30: Log is configured to *not* fsync() on all Append() calls
I20260925 03:02:01.213060 8302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07d98f0d1ef45fd8d262c9138c89e30: No bootstrap required, opened a new log
Sep 25 03:02:01 dist-test-slave-8zvk krb5kdc[8221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 03:02:01.224586 8302 raft_consensus.cc:348] T 00000000000000000000000000000000 P a07d98f0d1ef45fd8d262c9138c89e30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07d98f0d1ef45fd8d262c9138c89e30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } }
I20260925 03:02:01.225123 8302 raft_consensus.cc:374] T 00000000000000000000000000000000 P a07d98f0d1ef45fd8d262c9138c89e30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:01.225252 8302 raft_consensus.cc:729] T 00000000000000000000000000000000 P a07d98f0d1ef45fd8d262c9138c89e30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07d98f0d1ef45fd8d262c9138c89e30, State: Initialized, Role: FOLLOWER
I20260925 03:02:01.225920 8302 consensus_queue.cc:260] T 00000000000000000000000000000000 P a07d98f0d1ef45fd8d262c9138c89e30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07d98f0d1ef45fd8d262c9138c89e30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } }
I20260925 03:02:01.226156 8302 raft_consensus.cc:388] T 00000000000000000000000000000000 P a07d98f0d1ef45fd8d262c9138c89e30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:02:01.226301 8302 raft_consensus.cc:482] T 00000000000000000000000000000000 P a07d98f0d1ef45fd8d262c9138c89e30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:02:01.226492 8302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a07d98f0d1ef45fd8d262c9138c89e30 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:01.226672 8305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:01.202204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:59797 (local address 127.0.0.1:46595)
0925 03:02:01.203068 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:01.203082 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:01.203121 (+ 39us) server_negotiation.cc:407] Connection header received
0925 03:02:01.204722 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:01.204743 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:01.205134 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:01.205495 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:01.207292 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:01.208609 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:01.210373 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:01.211443 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:01.211557 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:01.216693 (+ 5136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:01.216704 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:01.216727 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:01.216776 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:01.220892 (+ 4116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:01.221856 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:01.221864 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:01.221873 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:01.222051 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:01.222640 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:01.222646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:01.222649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:01.223465 (+ 816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:01.223817 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:01.225181 (+ 1364us) server_negotiation.cc:299] Negotiation successful
0925 03:02:01.225537 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":390,"threads_started":1}
I20260925 03:02:01.228937 8302 raft_consensus.cc:504] T 00000000000000000000000000000000 P a07d98f0d1ef45fd8d262c9138c89e30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07d98f0d1ef45fd8d262c9138c89e30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } }
I20260925 03:02:01.229527 8302 leader_election.cc:302] T 00000000000000000000000000000000 P a07d98f0d1ef45fd8d262c9138c89e30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a07d98f0d1ef45fd8d262c9138c89e30; no voters:
I20260925 03:02:01.230222 8302 leader_election.cc:288] T 00000000000000000000000000000000 P a07d98f0d1ef45fd8d262c9138c89e30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:02:01.230341 8307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a07d98f0d1ef45fd8d262c9138c89e30 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:02:01.230427 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 03:02:01.232728 8302 sys_catalog.cc:564] T 00000000000000000000000000000000 P a07d98f0d1ef45fd8d262c9138c89e30 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:02:01.237479 8307 raft_consensus.cc:686] T 00000000000000000000000000000000 P a07d98f0d1ef45fd8d262c9138c89e30 [term 1 LEADER]: Becoming Leader. State: Replica: a07d98f0d1ef45fd8d262c9138c89e30, State: Running, Role: LEADER
I20260925 03:02:01.238173 8307 consensus_queue.cc:237] T 00000000000000000000000000000000 P a07d98f0d1ef45fd8d262c9138c89e30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07d98f0d1ef45fd8d262c9138c89e30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } }
I20260925 03:02:01.242769 8308 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07d98f0d1ef45fd8d262c9138c89e30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a07d98f0d1ef45fd8d262c9138c89e30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07d98f0d1ef45fd8d262c9138c89e30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } } }
I20260925 03:02:01.242913 8309 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07d98f0d1ef45fd8d262c9138c89e30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a07d98f0d1ef45fd8d262c9138c89e30. Latest consensus state: current_term: 1 leader_uuid: "a07d98f0d1ef45fd8d262c9138c89e30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07d98f0d1ef45fd8d262c9138c89e30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } } }
I20260925 03:02:01.243191 8308 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07d98f0d1ef45fd8d262c9138c89e30 [sys.catalog]: This master's current role is: LEADER
I20260925 03:02:01.243265 8309 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07d98f0d1ef45fd8d262c9138c89e30 [sys.catalog]: This master's current role is: LEADER
I20260925 03:02:01.244200 8322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:02:01.259092 8305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:01.235345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:57831 (local address 127.0.0.1:46595)
0925 03:02:01.235552 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:01.235560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:01.239445 (+ 3885us) server_negotiation.cc:407] Connection header received
0925 03:02:01.239676 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:01.239680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:01.239740 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:01.239842 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:01.241347 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:01.242736 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:01.244602 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:01.245444 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:01.245583 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:01.249656 (+ 4073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:01.249666 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:01.249674 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:01.249722 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:01.254210 (+ 4488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:01.255556 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:01.255565 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:01.255572 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:01.255748 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:01.256476 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:01.256485 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:01.256490 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:01.257030 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:01.257308 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:01.258521 (+ 1213us) server_negotiation.cc:299] Negotiation successful
0925 03:02:01.258705 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 03:02:01.261278 8322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:02:01.271034 8322 catalog_manager.cc:1347] Generated new cluster ID: e4a78d48dbd64597b719c8783968d392
I20260925 03:02:01.271145 8322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:02:01.299600 8322 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:02:01.300875 8322 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:02:01.306794 8322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a07d98f0d1ef45fd8d262c9138c89e30: Generated new TSK 0
I20260925 03:02:01.307683 8322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:02:01.321709 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 8234
I20260925 03:02:01.323237 8297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:02:01.419719 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 8234
W20260925 03:02:01.436916 8219 connection.cc:535] client connection to 127.0.0.1:46595 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46595): Connection reset by peer (error 104)
2026-09-25T03:02:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1970 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 03:02:01 dist-test-slave-8zvk krb5kdc[8334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 03:02:01 dist-test-slave-8zvk krb5kdc[8334](info): set up 2 sockets
Sep 25 03:02:01 dist-test-slave-8zvk krb5kdc[8334](info): commencing operation
krb5kdc: starting...
W20260925 03:02:02.518278 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 03:02:02 dist-test-slave-8zvk krb5kdc[8334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T03:02:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:02:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 03:02:02.753458 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33675
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:02:03.073060 8347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:02:03.073310 8347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:02:03.073498 8347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:02:03.080411 8347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:02:03.080531 8347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:02:03.080633 8347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:02:03.080734 8347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:02:03.080817 8347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:02:03.090260 8347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33675
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:02:03.091509 8347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:02:03.092799 8347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:02:03.104636 8355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:03.104663 8352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:03.104722 8353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:02:03.108290 8347 server_base.cc:1034] running on GCE node
I20260925 03:02:03.109079 8347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:02:03.110284 8347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:02:03.111521 8347 hybrid_clock.cc:648] HybridClock initialized: now 1790305323111493 us; error 40 us; skew 500 ppm
I20260925 03:02:03.111848 8347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:02:03 dist-test-slave-8zvk krb5kdc[8334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:02:03.114567 8347 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 03:02:03.115921 8347 webserver.cc:460] Webserver started at http://127.0.0.1:33431/ using document root <none> and password file <none>
I20260925 03:02:03.116513 8347 fs_manager.cc:359] Metadata directory not provided
I20260925 03:02:03.116658 8347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:02:03.116935 8347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:02:03.119731 8347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "28010ecaeab44d02b3a8de62816689cf"
format_stamp: "Formatted at 2026-09-25 03:02:03 on dist-test-slave-8zvk"
server_key: "b33304be0cd60e878fbb264c36bd734c"
server_key_iv: "6fea2a167467309922e15e7c9146c07f"
server_key_version: "encryptionkey@0"
I20260925 03:02:03.120412 8347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "28010ecaeab44d02b3a8de62816689cf"
format_stamp: "Formatted at 2026-09-25 03:02:03 on dist-test-slave-8zvk"
server_key: "b33304be0cd60e878fbb264c36bd734c"
server_key_iv: "6fea2a167467309922e15e7c9146c07f"
server_key_version: "encryptionkey@0"
I20260925 03:02:03.125041 8347 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 03:02:03.128229 8362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:03.129380 8347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 03:02:03.129544 8347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "28010ecaeab44d02b3a8de62816689cf"
format_stamp: "Formatted at 2026-09-25 03:02:03 on dist-test-slave-8zvk"
server_key: "b33304be0cd60e878fbb264c36bd734c"
server_key_iv: "6fea2a167467309922e15e7c9146c07f"
server_key_version: "encryptionkey@0"
I20260925 03:02:03.129719 8347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:02:03.144410 8347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:02:03.148003 8347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:02:03.148272 8347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:02:03.166097 8347 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33675
I20260925 03:02:03.166139 8413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33675 every 8 connection(s)
I20260925 03:02:03.167487 8347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:02:03.171073 8414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:03.174367 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 8347
I20260925 03:02:03.174542 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:02:03.175098 19219 external_mini_cluster.cc:1428] Setting key 99192e9426fc24ada5910c661c975966
I20260925 03:02:03.183208 8414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28010ecaeab44d02b3a8de62816689cf: Bootstrap starting.
I20260925 03:02:03.186764 8414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28010ecaeab44d02b3a8de62816689cf: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:03.187923 8414 log.cc:824] T 00000000000000000000000000000000 P 28010ecaeab44d02b3a8de62816689cf: Log is configured to *not* fsync() on all Append() calls
I20260925 03:02:03.188441 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:03.176797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:52123 (local address 127.0.0.1:33675)
0925 03:02:03.177558 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:03.177569 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:03.177608 (+ 39us) server_negotiation.cc:407] Connection header received
0925 03:02:03.179420 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:03.179441 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:03.179841 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:03.180148 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:03.180727 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:03.182052 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:03.183905 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:03.184550 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:03.184681 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:03.185975 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:03.185983 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:03.185994 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:02:03.186615 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:03.186699 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:02:03.186723 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:03.186897 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:03.187213 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 03:02:03.187388 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":344,"threads_started":1}
I20260925 03:02:03.191220 8414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28010ecaeab44d02b3a8de62816689cf: No bootstrap required, opened a new log
I20260925 03:02:03.192874 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 03:02:03.199826 8414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28010ecaeab44d02b3a8de62816689cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28010ecaeab44d02b3a8de62816689cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33675 } }
I20260925 03:02:03.200223 8414 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28010ecaeab44d02b3a8de62816689cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:03.200351 8414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28010ecaeab44d02b3a8de62816689cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28010ecaeab44d02b3a8de62816689cf, State: Initialized, Role: FOLLOWER
I20260925 03:02:03.200970 8414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28010ecaeab44d02b3a8de62816689cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28010ecaeab44d02b3a8de62816689cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33675 } }
I20260925 03:02:03.201238 8414 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28010ecaeab44d02b3a8de62816689cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:02:03.201396 8414 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28010ecaeab44d02b3a8de62816689cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:02:03.201577 8414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28010ecaeab44d02b3a8de62816689cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:03.203903 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:03.197123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:40755 (local address 127.0.0.1:33675)
0925 03:02:03.197283 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:03.197287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:03.197786 (+ 499us) server_negotiation.cc:407] Connection header received
0925 03:02:03.198031 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:03.198035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:03.198101 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:03.198213 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:03.198752 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:03.199566 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:03.201000 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:03.201624 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:03.201771 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:03.203158 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:03.203166 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:03.203171 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:02:03.203191 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:03.203255 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:02:03.203274 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:03.203402 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:03.203522 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 03:02:03.203574 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 03:02:03.204635 8414 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28010ecaeab44d02b3a8de62816689cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28010ecaeab44d02b3a8de62816689cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33675 } }
I20260925 03:02:03.205193 8414 leader_election.cc:302] T 00000000000000000000000000000000 P 28010ecaeab44d02b3a8de62816689cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28010ecaeab44d02b3a8de62816689cf; no voters:
I20260925 03:02:03.205750 8414 leader_election.cc:288] T 00000000000000000000000000000000 P 28010ecaeab44d02b3a8de62816689cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:02:03.205873 8424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28010ecaeab44d02b3a8de62816689cf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:02:03.206637 8424 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28010ecaeab44d02b3a8de62816689cf [term 1 LEADER]: Becoming Leader. State: Replica: 28010ecaeab44d02b3a8de62816689cf, State: Running, Role: LEADER
I20260925 03:02:03.207178 8424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28010ecaeab44d02b3a8de62816689cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28010ecaeab44d02b3a8de62816689cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33675 } }
I20260925 03:02:03.207876 8414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28010ecaeab44d02b3a8de62816689cf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:02:03.211304 8426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28010ecaeab44d02b3a8de62816689cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28010ecaeab44d02b3a8de62816689cf. Latest consensus state: current_term: 1 leader_uuid: "28010ecaeab44d02b3a8de62816689cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28010ecaeab44d02b3a8de62816689cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33675 } } }
I20260925 03:02:03.211715 8426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28010ecaeab44d02b3a8de62816689cf [sys.catalog]: This master's current role is: LEADER
I20260925 03:02:03.212349 8425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28010ecaeab44d02b3a8de62816689cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28010ecaeab44d02b3a8de62816689cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28010ecaeab44d02b3a8de62816689cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33675 } } }
I20260925 03:02:03.212615 8425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28010ecaeab44d02b3a8de62816689cf [sys.catalog]: This master's current role is: LEADER
I20260925 03:02:03.219956 8433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:02:03.227182 8433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:02:03.237772 8433 catalog_manager.cc:1347] Generated new cluster ID: 89c57cd5de294da6a8da4d9f9dfb1a67
I20260925 03:02:03.237890 8433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:02:03.273425 8433 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:02:03.275231 8433 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:02:03.292332 8433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28010ecaeab44d02b3a8de62816689cf: Generated new TSK 0
I20260925 03:02:03.293449 8433 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:02:03.303059 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 8347
W20260925 03:02:03.304991 8409 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.55:52123
W20260925 03:02:03.305686 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 8347: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 03:02:03.305827 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 8347
W20260925 03:02:03.323391 8332 connection.cc:535] client connection to 127.0.0.1:33675 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33675): Connection reset by peer (error 104)
2026-09-25T03:02:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1886 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 03:02:03 dist-test-slave-8zvk krb5kdc[8445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 03:02:03 dist-test-slave-8zvk krb5kdc[8445](info): set up 2 sockets
Sep 25 03:02:03 dist-test-slave-8zvk krb5kdc[8445](info): commencing operation
krb5kdc: starting...
W20260925 03:02:04.406638 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 03:02:04 dist-test-slave-8zvk krb5kdc[8445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T03:02:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:02:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 03:02:04.680220 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46199
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:02:05.015105 8458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:02:05.015362 8458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:02:05.015559 8458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:02:05.022706 8458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:02:05.022809 8458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:02:05.022953 8458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:02:05.023054 8458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:02:05.023130 8458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:02:05.032706 8458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46199
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:02:05.033921 8458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:02:05.035154 8458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:02:05.046859 8466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:05.046942 8464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:05.047076 8463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:02:05.047755 8458 server_base.cc:1034] running on GCE node
I20260925 03:02:05.048550 8458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:02:05.049886 8458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:02:05.051110 8458 hybrid_clock.cc:648] HybridClock initialized: now 1790305325051069 us; error 54 us; skew 500 ppm
I20260925 03:02:05.051450 8458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:02:05 dist-test-slave-8zvk krb5kdc[8445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:02:05.069819 8458 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 03:02:05.071448 8458 webserver.cc:460] Webserver started at http://127.0.0.1:34059/ using document root <none> and password file <none>
I20260925 03:02:05.072170 8458 fs_manager.cc:359] Metadata directory not provided
I20260925 03:02:05.072325 8458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:02:05.072644 8458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:02:05.075785 8458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "332c9f16dd1240a5a84d473617f5a110"
format_stamp: "Formatted at 2026-09-25 03:02:05 on dist-test-slave-8zvk"
server_key: "56cc333b9d2b9a4054eb6880f8551be7"
server_key_iv: "0b626fe43721ab23c21e4599f33a824e"
server_key_version: "encryptionkey@0"
I20260925 03:02:05.076628 8458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "332c9f16dd1240a5a84d473617f5a110"
format_stamp: "Formatted at 2026-09-25 03:02:05 on dist-test-slave-8zvk"
server_key: "56cc333b9d2b9a4054eb6880f8551be7"
server_key_iv: "0b626fe43721ab23c21e4599f33a824e"
server_key_version: "encryptionkey@0"
I20260925 03:02:05.081779 8458 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 03:02:05.085440 8473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:05.087162 8458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 03:02:05.087360 8458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "332c9f16dd1240a5a84d473617f5a110"
format_stamp: "Formatted at 2026-09-25 03:02:05 on dist-test-slave-8zvk"
server_key: "56cc333b9d2b9a4054eb6880f8551be7"
server_key_iv: "0b626fe43721ab23c21e4599f33a824e"
server_key_version: "encryptionkey@0"
I20260925 03:02:05.087569 8458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:02:05.093662 8458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:02:05.098130 8458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:02:05.098451 8458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:02:05.117295 8458 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46199
I20260925 03:02:05.117341 8524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46199 every 8 connection(s)
I20260925 03:02:05.118736 8458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:02:05.122318 8525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:05.127310 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 8458
I20260925 03:02:05.127525 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:02:05.128057 19219 external_mini_cluster.cc:1428] Setting key 7ce61911b701b06a7ec142aad27f31cd
I20260925 03:02:05.133731 8525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 332c9f16dd1240a5a84d473617f5a110: Bootstrap starting.
I20260925 03:02:05.135741 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:05.129807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:52775 (local address 127.0.0.1:46199)
0925 03:02:05.130622 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:05.130633 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:05.130673 (+ 40us) server_negotiation.cc:407] Connection header received
0925 03:02:05.132439 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:05.132460 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:05.132775 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:05.133070 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:05.134132 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:05.134138 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:05.134150 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:02:05.134170 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:05.134228 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:02:05.134247 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:05.134412 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:05.134580 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 03:02:05.134796 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":359,"threads_started":1}
I20260925 03:02:05.136993 8525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 332c9f16dd1240a5a84d473617f5a110: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:05.138074 8525 log.cc:824] T 00000000000000000000000000000000 P 332c9f16dd1240a5a84d473617f5a110: Log is configured to *not* fsync() on all Append() calls
I20260925 03:02:05.139391 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 03:02:05.140821 8525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 332c9f16dd1240a5a84d473617f5a110: No bootstrap required, opened a new log
I20260925 03:02:05.146467 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:05.143817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:34329 (local address 127.0.0.1:46199)
0925 03:02:05.143972 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:05.143976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:05.144417 (+ 441us) server_negotiation.cc:407] Connection header received
0925 03:02:05.144616 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:05.144620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:05.144673 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:05.144785 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:05.145751 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:05.145760 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:05.145765 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:02:05.145794 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:05.145833 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:02:05.145848 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:05.145983 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:05.146122 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 03:02:05.146192 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 03:02:05.149329 8525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 332c9f16dd1240a5a84d473617f5a110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332c9f16dd1240a5a84d473617f5a110" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46199 } }
I20260925 03:02:05.149720 8525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 332c9f16dd1240a5a84d473617f5a110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:05.149832 8525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 332c9f16dd1240a5a84d473617f5a110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 332c9f16dd1240a5a84d473617f5a110, State: Initialized, Role: FOLLOWER
I20260925 03:02:05.150374 8525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 332c9f16dd1240a5a84d473617f5a110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332c9f16dd1240a5a84d473617f5a110" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46199 } }
I20260925 03:02:05.150633 8525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 332c9f16dd1240a5a84d473617f5a110 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:02:05.150779 8525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 332c9f16dd1240a5a84d473617f5a110 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:02:05.150977 8525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 332c9f16dd1240a5a84d473617f5a110 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:05.153308 8525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 332c9f16dd1240a5a84d473617f5a110 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332c9f16dd1240a5a84d473617f5a110" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46199 } }
I20260925 03:02:05.153898 8525 leader_election.cc:302] T 00000000000000000000000000000000 P 332c9f16dd1240a5a84d473617f5a110 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 332c9f16dd1240a5a84d473617f5a110; no voters:
I20260925 03:02:05.154587 8525 leader_election.cc:288] T 00000000000000000000000000000000 P 332c9f16dd1240a5a84d473617f5a110 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:02:05.154693 8535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 332c9f16dd1240a5a84d473617f5a110 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:02:05.155362 8535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 332c9f16dd1240a5a84d473617f5a110 [term 1 LEADER]: Becoming Leader. State: Replica: 332c9f16dd1240a5a84d473617f5a110, State: Running, Role: LEADER
I20260925 03:02:05.155961 8535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 332c9f16dd1240a5a84d473617f5a110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332c9f16dd1240a5a84d473617f5a110" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46199 } }
I20260925 03:02:05.156858 8525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 332c9f16dd1240a5a84d473617f5a110 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:02:05.159171 8536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 332c9f16dd1240a5a84d473617f5a110 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "332c9f16dd1240a5a84d473617f5a110" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332c9f16dd1240a5a84d473617f5a110" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46199 } } }
I20260925 03:02:05.159695 8536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 332c9f16dd1240a5a84d473617f5a110 [sys.catalog]: This master's current role is: LEADER
I20260925 03:02:05.159556 8537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 332c9f16dd1240a5a84d473617f5a110 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 332c9f16dd1240a5a84d473617f5a110. Latest consensus state: current_term: 1 leader_uuid: "332c9f16dd1240a5a84d473617f5a110" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332c9f16dd1240a5a84d473617f5a110" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46199 } } }
I20260925 03:02:05.159852 8537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 332c9f16dd1240a5a84d473617f5a110 [sys.catalog]: This master's current role is: LEADER
I20260925 03:02:05.167917 8542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:02:05.173498 8542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:02:05.182037 8542 catalog_manager.cc:1347] Generated new cluster ID: a0b6efc54b774d6cad6e8d64d96ca88e
I20260925 03:02:05.182144 8542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:02:05.209504 8542 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:02:05.210005 8542 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:02:05.223801 8542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 332c9f16dd1240a5a84d473617f5a110: Generated new TSK 0
I20260925 03:02:05.224957 8542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:02:05.239251 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 8458
W20260925 03:02:05.240672 8520 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.55:52775
W20260925 03:02:05.241595 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 8458: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 03:02:05.241768 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 8458
2026-09-25T03:02:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1936 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 03:02:05 dist-test-slave-8zvk krb5kdc[8556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 03:02:05 dist-test-slave-8zvk krb5kdc[8556](info): set up 2 sockets
Sep 25 03:02:05 dist-test-slave-8zvk krb5kdc[8556](info): commencing operation
krb5kdc: starting...
W20260925 03:02:06.330298 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 03:02:06 dist-test-slave-8zvk krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T03:02:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:02:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 03:02:06.581775 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:42493
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:36127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:42493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:02:06.908244 8569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:02:06.908556 8569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:02:06.908727 8569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:02:06.916139 8569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:02:06.916268 8569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:02:06.916373 8569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:02:06.916487 8569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:02:06.916574 8569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:02:06.926246 8569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:36127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:42493
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:42493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:02:06.927587 8569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:02:06.928894 8569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:02:06.940342 8577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:06.940384 8575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:06.940456 8574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:02:06.941000 8569 server_base.cc:1034] running on GCE node
I20260925 03:02:06.941808 8569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:02:06.943301 8569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:02:06.944551 8569 hybrid_clock.cc:648] HybridClock initialized: now 1790305326944509 us; error 52 us; skew 500 ppm
I20260925 03:02:06.944901 8569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:02:06 dist-test-slave-8zvk krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305326, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:02:06.947955 8569 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 03:02:06.949357 8569 webserver.cc:460] Webserver started at http://127.18.196.254:38959/ using document root <none> and password file <none>
I20260925 03:02:06.950019 8569 fs_manager.cc:359] Metadata directory not provided
I20260925 03:02:06.950166 8569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:02:06.950482 8569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:02:06.953606 8569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "5862a5d199b94c88a281ec1a1ff9c75d"
format_stamp: "Formatted at 2026-09-25 03:02:06 on dist-test-slave-8zvk"
server_key: "7ee584ffff9951af417314dfa3a80a63"
server_key_iv: "f7624e6e7ed7f5f62805ef6e8b56caae"
server_key_version: "encryptionkey@0"
I20260925 03:02:06.954365 8569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "5862a5d199b94c88a281ec1a1ff9c75d"
format_stamp: "Formatted at 2026-09-25 03:02:06 on dist-test-slave-8zvk"
server_key: "7ee584ffff9951af417314dfa3a80a63"
server_key_iv: "f7624e6e7ed7f5f62805ef6e8b56caae"
server_key_version: "encryptionkey@0"
I20260925 03:02:06.959554 8569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 03:02:06.962713 8584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:06.963973 8569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 03:02:06.964176 8569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "5862a5d199b94c88a281ec1a1ff9c75d"
format_stamp: "Formatted at 2026-09-25 03:02:06 on dist-test-slave-8zvk"
server_key: "7ee584ffff9951af417314dfa3a80a63"
server_key_iv: "f7624e6e7ed7f5f62805ef6e8b56caae"
server_key_version: "encryptionkey@0"
I20260925 03:02:06.964386 8569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:02:06.988102 8569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:02:06.991853 8569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:02:06.992142 8569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:02:07.010291 8569 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:42493
I20260925 03:02:07.010314 8635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:42493 every 8 connection(s)
I20260925 03:02:07.011586 8569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:02:07.014022 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 8569
I20260925 03:02:07.014202 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:02:07.014628 19219 external_mini_cluster.cc:1428] Setting key 54cfaed5d5b37b856b593ef589822049
I20260925 03:02:07.015228 8636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 03:02:07 dist-test-slave-8zvk krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:02:07.029268 8636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5862a5d199b94c88a281ec1a1ff9c75d: Bootstrap starting.
I20260925 03:02:07.033318 8636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5862a5d199b94c88a281ec1a1ff9c75d: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:07.034461 8636 log.cc:824] T 00000000000000000000000000000000 P 5862a5d199b94c88a281ec1a1ff9c75d: Log is configured to *not* fsync() on all Append() calls
I20260925 03:02:07.037774 8636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5862a5d199b94c88a281ec1a1ff9c75d: No bootstrap required, opened a new log
I20260925 03:02:07.038339 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:07.016284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53987 (local address 127.18.196.254:42493)
0925 03:02:07.017069 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:07.017084 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:07.017131 (+ 47us) server_negotiation.cc:407] Connection header received
0925 03:02:07.018967 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:07.018994 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:07.019503 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:07.019932 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:07.021207 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:07.023282 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:07.024484 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:07.025485 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:07.025675 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:07.029889 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:07.029899 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:07.029915 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:07.029959 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:07.033859 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:07.034622 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:07.034630 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:07.034636 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:07.034829 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:07.035402 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:07.035405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:07.035410 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:07.035979 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:07.036290 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:07.036707 (+ 417us) server_negotiation.cc:299] Negotiation successful
0925 03:02:07.037030 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":331,"threads_started":1}
I20260925 03:02:07.042189 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 03:02:07.046083 8636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5862a5d199b94c88a281ec1a1ff9c75d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5862a5d199b94c88a281ec1a1ff9c75d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 42493 } }
I20260925 03:02:07.046571 8636 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5862a5d199b94c88a281ec1a1ff9c75d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:07.046775 8636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5862a5d199b94c88a281ec1a1ff9c75d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5862a5d199b94c88a281ec1a1ff9c75d, State: Initialized, Role: FOLLOWER
I20260925 03:02:07.047415 8636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5862a5d199b94c88a281ec1a1ff9c75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5862a5d199b94c88a281ec1a1ff9c75d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 42493 } }
I20260925 03:02:07.047658 8636 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5862a5d199b94c88a281ec1a1ff9c75d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:02:07.047847 8636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5862a5d199b94c88a281ec1a1ff9c75d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:02:07.048025 8636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5862a5d199b94c88a281ec1a1ff9c75d [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:07.050195 8636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5862a5d199b94c88a281ec1a1ff9c75d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5862a5d199b94c88a281ec1a1ff9c75d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 42493 } }
I20260925 03:02:07.050676 8636 leader_election.cc:302] T 00000000000000000000000000000000 P 5862a5d199b94c88a281ec1a1ff9c75d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5862a5d199b94c88a281ec1a1ff9c75d; no voters:
I20260925 03:02:07.051403 8636 leader_election.cc:288] T 00000000000000000000000000000000 P 5862a5d199b94c88a281ec1a1ff9c75d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:02:07.051513 8646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5862a5d199b94c88a281ec1a1ff9c75d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:02:07.053115 8646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5862a5d199b94c88a281ec1a1ff9c75d [term 1 LEADER]: Becoming Leader. State: Replica: 5862a5d199b94c88a281ec1a1ff9c75d, State: Running, Role: LEADER
I20260925 03:02:07.053642 8646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5862a5d199b94c88a281ec1a1ff9c75d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5862a5d199b94c88a281ec1a1ff9c75d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 42493 } }
I20260925 03:02:07.053794 8636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5862a5d199b94c88a281ec1a1ff9c75d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:02:07.056690 8647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5862a5d199b94c88a281ec1a1ff9c75d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5862a5d199b94c88a281ec1a1ff9c75d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5862a5d199b94c88a281ec1a1ff9c75d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 42493 } } }
I20260925 03:02:07.057796 8647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5862a5d199b94c88a281ec1a1ff9c75d [sys.catalog]: This master's current role is: LEADER
I20260925 03:02:07.057436 8648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5862a5d199b94c88a281ec1a1ff9c75d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5862a5d199b94c88a281ec1a1ff9c75d. Latest consensus state: current_term: 1 leader_uuid: "5862a5d199b94c88a281ec1a1ff9c75d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5862a5d199b94c88a281ec1a1ff9c75d" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 42493 } } }
I20260925 03:02:07.058395 8648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5862a5d199b94c88a281ec1a1ff9c75d [sys.catalog]: This master's current role is: LEADER
I20260925 03:02:07.063488 8652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:02:07.063943 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:07.046586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37291 (local address 127.18.196.254:42493)
0925 03:02:07.046773 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:07.046779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:07.047164 (+ 385us) server_negotiation.cc:407] Connection header received
0925 03:02:07.047374 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:07.047378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:07.047436 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:07.047537 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:07.049036 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:07.049865 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:07.051315 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:07.052067 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:07.052204 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:07.054994 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:07.054999 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:07.055002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:07.055036 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:07.059301 (+ 4265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:07.060565 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:07.060573 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:07.060579 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:07.060753 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:07.061499 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:07.061506 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:07.061516 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:07.062033 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:07.063126 (+ 1093us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:07.063283 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 03:02:07.063413 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 03:02:07.070336 8652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:02:07.081125 8652 catalog_manager.cc:1347] Generated new cluster ID: 24a7885f3b0c4b9396d94ca2157373dd
I20260925 03:02:07.081245 8652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:02:07.108811 8652 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:02:07.110222 8652 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:02:07.117791 8652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5862a5d199b94c88a281ec1a1ff9c75d: Generated new TSK 0
I20260925 03:02:07.118758 8652 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:02:07.125764 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 8569
I20260925 03:02:07.127472 8631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:02:07.222033 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 8569
W20260925 03:02:07.239101 8554 connection.cc:535] client connection to 127.18.196.254:42493 recv error: Network error: failed to read from TLS socket (remote: 127.18.196.254:42493): Connection reset by peer (error 104)
2026-09-25T03:02:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1979 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 03:02:07 dist-test-slave-8zvk krb5kdc[8668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 03:02:07 dist-test-slave-8zvk krb5kdc[8668](info): set up 2 sockets
Sep 25 03:02:07 dist-test-slave-8zvk krb5kdc[8668](info): commencing operation
krb5kdc: starting...
W20260925 03:02:08.310400 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 03:02:08 dist-test-slave-8zvk krb5kdc[8668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T03:02:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:02:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 03:02:08.559907 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:35485
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:41979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:35485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:02:08.878147 8681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:02:08.878453 8681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:02:08.878628 8681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:02:08.885087 8681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:02:08.885217 8681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:02:08.885324 8681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:02:08.885428 8681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:02:08.885491 8681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:02:08.894532 8681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:41979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:35485
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:35485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:02:08.895777 8681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:02:08.897056 8681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:02:08.908818 8686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:08.908885 8689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:08.908789 8687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:02:08.909749 8681 server_base.cc:1034] running on GCE node
I20260925 03:02:08.910557 8681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:02:08.911865 8681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:02:08.913093 8681 hybrid_clock.cc:648] HybridClock initialized: now 1790305328913047 us; error 51 us; skew 500 ppm
I20260925 03:02:08.913424 8681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:02:08 dist-test-slave-8zvk krb5kdc[8668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305328, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:02:08.916486 8681 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 03:02:08.917886 8681 webserver.cc:460] Webserver started at http://127.18.196.254:33263/ using document root <none> and password file <none>
I20260925 03:02:08.918524 8681 fs_manager.cc:359] Metadata directory not provided
I20260925 03:02:08.918673 8681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:02:08.919020 8681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:02:08.921856 8681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "e0a75980e33f4b6db44c521025cad662"
format_stamp: "Formatted at 2026-09-25 03:02:08 on dist-test-slave-8zvk"
server_key: "32d7ebb3599f4644d85ec81edcf58dce"
server_key_iv: "c2eb19ac00b1b7e0a2989baa8aec6c48"
server_key_version: "encryptionkey@0"
I20260925 03:02:08.922616 8681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "e0a75980e33f4b6db44c521025cad662"
format_stamp: "Formatted at 2026-09-25 03:02:08 on dist-test-slave-8zvk"
server_key: "32d7ebb3599f4644d85ec81edcf58dce"
server_key_iv: "c2eb19ac00b1b7e0a2989baa8aec6c48"
server_key_version: "encryptionkey@0"
I20260925 03:02:08.927399 8681 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260925 03:02:08.930615 8696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:08.931896 8681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 03:02:08.932078 8681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "e0a75980e33f4b6db44c521025cad662"
format_stamp: "Formatted at 2026-09-25 03:02:08 on dist-test-slave-8zvk"
server_key: "32d7ebb3599f4644d85ec81edcf58dce"
server_key_iv: "c2eb19ac00b1b7e0a2989baa8aec6c48"
server_key_version: "encryptionkey@0"
I20260925 03:02:08.932271 8681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:02:08.945778 8681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:02:08.949805 8681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:02:08.950088 8681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:02:08.967444 8681 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:35485
I20260925 03:02:08.967476 8747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:35485 every 8 connection(s)
I20260925 03:02:08.968673 8681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:02:08.971189 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 8681
I20260925 03:02:08.971361 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:02:08.971877 19219 external_mini_cluster.cc:1428] Setting key 18fdc19973b56c6ef274e234f6dfa7e4
I20260925 03:02:08.972362 8748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:08.984694 8751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:08.973457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43505 (local address 127.18.196.254:35485)
0925 03:02:08.974204 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:08.974213 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:08.974251 (+ 38us) server_negotiation.cc:407] Connection header received
0925 03:02:08.976435 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:08.976454 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:08.976777 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:08.977064 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:08.977640 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:08.979013 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:08.980687 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:08.981267 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:08.981396 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:08.982600 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:08.982607 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:08.982618 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:02:08.983212 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:08.983283 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:02:08.983309 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:08.983420 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:08.983700 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 03:02:08.983873 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":393,"threads_started":1}
I20260925 03:02:08.984921 8748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0a75980e33f4b6db44c521025cad662: Bootstrap starting.
I20260925 03:02:08.988086 8748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0a75980e33f4b6db44c521025cad662: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:08.988240 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 03:02:08.989068 8748 log.cc:824] T 00000000000000000000000000000000 P e0a75980e33f4b6db44c521025cad662: Log is configured to *not* fsync() on all Append() calls
I20260925 03:02:08.991588 8748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0a75980e33f4b6db44c521025cad662: No bootstrap required, opened a new log
I20260925 03:02:09.000699 8748 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0a75980e33f4b6db44c521025cad662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a75980e33f4b6db44c521025cad662" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 35485 } }
I20260925 03:02:09.001187 8748 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0a75980e33f4b6db44c521025cad662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:09.001338 8748 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0a75980e33f4b6db44c521025cad662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0a75980e33f4b6db44c521025cad662, State: Initialized, Role: FOLLOWER
I20260925 03:02:09.001533 8751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:08.992965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38829 (local address 127.18.196.254:35485)
0925 03:02:08.993163 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:08.993170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:08.995408 (+ 2238us) server_negotiation.cc:407] Connection header received
0925 03:02:08.995590 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:08.995594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:08.995653 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:08.995735 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:08.996349 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:08.997101 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:08.999003 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:08.999445 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:08.999534 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:09.000712 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:09.000716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:09.000719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:02:09.000735 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:09.000769 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:02:09.000782 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:09.000866 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:09.001128 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 03:02:09.001181 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 03:02:09.002022 8748 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0a75980e33f4b6db44c521025cad662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a75980e33f4b6db44c521025cad662" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 35485 } }
I20260925 03:02:09.002270 8748 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0a75980e33f4b6db44c521025cad662 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:02:09.002434 8748 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0a75980e33f4b6db44c521025cad662 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:02:09.002628 8748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0a75980e33f4b6db44c521025cad662 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:09.004853 8748 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0a75980e33f4b6db44c521025cad662 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a75980e33f4b6db44c521025cad662" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 35485 } }
I20260925 03:02:09.005330 8748 leader_election.cc:302] T 00000000000000000000000000000000 P e0a75980e33f4b6db44c521025cad662 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0a75980e33f4b6db44c521025cad662; no voters:
I20260925 03:02:09.005852 8748 leader_election.cc:288] T 00000000000000000000000000000000 P e0a75980e33f4b6db44c521025cad662 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:02:09.005990 8758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0a75980e33f4b6db44c521025cad662 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:02:09.006685 8758 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0a75980e33f4b6db44c521025cad662 [term 1 LEADER]: Becoming Leader. State: Replica: e0a75980e33f4b6db44c521025cad662, State: Running, Role: LEADER
I20260925 03:02:09.007212 8758 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0a75980e33f4b6db44c521025cad662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a75980e33f4b6db44c521025cad662" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 35485 } }
I20260925 03:02:09.007877 8748 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0a75980e33f4b6db44c521025cad662 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:02:09.010306 8759 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0a75980e33f4b6db44c521025cad662 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0a75980e33f4b6db44c521025cad662" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a75980e33f4b6db44c521025cad662" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 35485 } } }
I20260925 03:02:09.010727 8759 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0a75980e33f4b6db44c521025cad662 [sys.catalog]: This master's current role is: LEADER
I20260925 03:02:09.010330 8760 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0a75980e33f4b6db44c521025cad662 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0a75980e33f4b6db44c521025cad662. Latest consensus state: current_term: 1 leader_uuid: "e0a75980e33f4b6db44c521025cad662" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a75980e33f4b6db44c521025cad662" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 35485 } } }
I20260925 03:02:09.011999 8760 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0a75980e33f4b6db44c521025cad662 [sys.catalog]: This master's current role is: LEADER
I20260925 03:02:09.013024 8764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:02:09.017879 8764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:02:09.025591 8764 catalog_manager.cc:1347] Generated new cluster ID: d0fb29abdc99429f9d9716801b2bfef2
I20260925 03:02:09.025686 8764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:02:09.045676 8764 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:02:09.047403 8764 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:02:09.062892 8764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0a75980e33f4b6db44c521025cad662: Generated new TSK 0
I20260925 03:02:09.063621 8764 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:02:09.080861 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 8681
W20260925 03:02:09.082098 8743 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43505
W20260925 03:02:09.082897 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 8681: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 03:02:09.083071 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 8681
W20260925 03:02:09.100171 8666 connection.cc:535] client connection to 127.18.196.254:35485 recv error: Network error: failed to read from TLS socket (remote: 127.18.196.254:35485): Connection reset by peer (error 104)
2026-09-25T03:02:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1861 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 03:02:09 dist-test-slave-8zvk krb5kdc[8779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 03:02:09 dist-test-slave-8zvk krb5kdc[8779](info): set up 2 sockets
Sep 25 03:02:09 dist-test-slave-8zvk krb5kdc[8779](info): commencing operation
krb5kdc: starting...
W20260925 03:02:10.174401 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 03:02:10 dist-test-slave-8zvk krb5kdc[8779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T03:02:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:02:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 03:02:10.421476 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:35707
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:38277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:35707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:02:10.738454 8792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:02:10.738713 8792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:02:10.738801 8792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:02:10.745735 8792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:02:10.745850 8792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:02:10.745915 8792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:02:10.745996 8792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:02:10.746033 8792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:02:10.755198 8792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:38277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:35707
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:35707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:02:10.756283 8792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:02:10.757385 8792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:02:10.768707 8800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:10.768795 8797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:10.768764 8798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:02:10.769372 8792 server_base.cc:1034] running on GCE node
I20260925 03:02:10.770138 8792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:02:10.771554 8792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:02:10.772794 8792 hybrid_clock.cc:648] HybridClock initialized: now 1790305330772734 us; error 47 us; skew 500 ppm
I20260925 03:02:10.773154 8792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:02:10 dist-test-slave-8zvk krb5kdc[8779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305330, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:02:10.780651 8792 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 03:02:10.782100 8792 webserver.cc:460] Webserver started at http://127.18.196.254:46425/ using document root <none> and password file <none>
I20260925 03:02:10.782752 8792 fs_manager.cc:359] Metadata directory not provided
I20260925 03:02:10.782914 8792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:02:10.783226 8792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:02:10.786134 8792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "c30d606c6a834429bef8d6fe9d462fbe"
format_stamp: "Formatted at 2026-09-25 03:02:10 on dist-test-slave-8zvk"
server_key: "4038199c87587388c1eb19d1a5f8414d"
server_key_iv: "5c310b2ad540803f9e17dd433ee621c6"
server_key_version: "encryptionkey@0"
I20260925 03:02:10.786986 8792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "c30d606c6a834429bef8d6fe9d462fbe"
format_stamp: "Formatted at 2026-09-25 03:02:10 on dist-test-slave-8zvk"
server_key: "4038199c87587388c1eb19d1a5f8414d"
server_key_iv: "5c310b2ad540803f9e17dd433ee621c6"
server_key_version: "encryptionkey@0"
I20260925 03:02:10.791846 8792 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 03:02:10.794796 8807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:10.796026 8792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 03:02:10.796218 8792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "c30d606c6a834429bef8d6fe9d462fbe"
format_stamp: "Formatted at 2026-09-25 03:02:10 on dist-test-slave-8zvk"
server_key: "4038199c87587388c1eb19d1a5f8414d"
server_key_iv: "5c310b2ad540803f9e17dd433ee621c6"
server_key_version: "encryptionkey@0"
I20260925 03:02:10.796406 8792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:02:10.801607 8792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:02:10.805519 8792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:02:10.805800 8792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:02:10.824347 8792 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:35707
I20260925 03:02:10.824389 8858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:35707 every 8 connection(s)
I20260925 03:02:10.825791 8792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:02:10.829546 8859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:10.832592 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 8792
I20260925 03:02:10.832765 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:02:10.833315 19219 external_mini_cluster.cc:1428] Setting key 6a1233b6ad7259a2ebc133fb8fd26b67
I20260925 03:02:10.841149 8862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:10.835240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52907 (local address 127.18.196.254:35707)
0925 03:02:10.836029 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:10.836041 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:10.836137 (+ 96us) server_negotiation.cc:407] Connection header received
0925 03:02:10.837822 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:10.837845 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:10.838209 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:10.838534 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:10.839632 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:10.839638 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:10.839651 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:02:10.839667 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:10.839725 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:02:10.839744 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:10.839919 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:10.840019 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 03:02:10.840239 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":361,"threads_started":1}
I20260925 03:02:10.842005 8859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c30d606c6a834429bef8d6fe9d462fbe: Bootstrap starting.
I20260925 03:02:10.844698 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 03:02:10.845188 8859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c30d606c6a834429bef8d6fe9d462fbe: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:10.846410 8859 log.cc:824] T 00000000000000000000000000000000 P c30d606c6a834429bef8d6fe9d462fbe: Log is configured to *not* fsync() on all Append() calls
I20260925 03:02:10.849699 8859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c30d606c6a834429bef8d6fe9d462fbe: No bootstrap required, opened a new log
I20260925 03:02:10.852380 8862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:10.849923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47871 (local address 127.18.196.254:35707)
0925 03:02:10.850129 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:10.850134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:10.850455 (+ 321us) server_negotiation.cc:407] Connection header received
0925 03:02:10.850625 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:10.850628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:10.850668 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:10.850741 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:10.851716 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:10.851720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:10.851723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:02:10.851740 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:10.851775 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:02:10.851788 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:10.851896 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:10.852026 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 03:02:10.852077 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 03:02:10.859413 8859 raft_consensus.cc:348] T 00000000000000000000000000000000 P c30d606c6a834429bef8d6fe9d462fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30d606c6a834429bef8d6fe9d462fbe" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 35707 } }
I20260925 03:02:10.859772 8859 raft_consensus.cc:374] T 00000000000000000000000000000000 P c30d606c6a834429bef8d6fe9d462fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:10.859896 8859 raft_consensus.cc:729] T 00000000000000000000000000000000 P c30d606c6a834429bef8d6fe9d462fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30d606c6a834429bef8d6fe9d462fbe, State: Initialized, Role: FOLLOWER
I20260925 03:02:10.860507 8859 consensus_queue.cc:260] T 00000000000000000000000000000000 P c30d606c6a834429bef8d6fe9d462fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30d606c6a834429bef8d6fe9d462fbe" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 35707 } }
I20260925 03:02:10.860744 8859 raft_consensus.cc:388] T 00000000000000000000000000000000 P c30d606c6a834429bef8d6fe9d462fbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:02:10.860929 8859 raft_consensus.cc:482] T 00000000000000000000000000000000 P c30d606c6a834429bef8d6fe9d462fbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:02:10.861107 8859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c30d606c6a834429bef8d6fe9d462fbe [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:10.863380 8859 raft_consensus.cc:504] T 00000000000000000000000000000000 P c30d606c6a834429bef8d6fe9d462fbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30d606c6a834429bef8d6fe9d462fbe" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 35707 } }
I20260925 03:02:10.863863 8859 leader_election.cc:302] T 00000000000000000000000000000000 P c30d606c6a834429bef8d6fe9d462fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c30d606c6a834429bef8d6fe9d462fbe; no voters:
I20260925 03:02:10.864431 8859 leader_election.cc:288] T 00000000000000000000000000000000 P c30d606c6a834429bef8d6fe9d462fbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:02:10.864593 8869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c30d606c6a834429bef8d6fe9d462fbe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:02:10.865267 8869 raft_consensus.cc:686] T 00000000000000000000000000000000 P c30d606c6a834429bef8d6fe9d462fbe [term 1 LEADER]: Becoming Leader. State: Replica: c30d606c6a834429bef8d6fe9d462fbe, State: Running, Role: LEADER
I20260925 03:02:10.865810 8869 consensus_queue.cc:237] T 00000000000000000000000000000000 P c30d606c6a834429bef8d6fe9d462fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30d606c6a834429bef8d6fe9d462fbe" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 35707 } }
I20260925 03:02:10.866521 8859 sys_catalog.cc:564] T 00000000000000000000000000000000 P c30d606c6a834429bef8d6fe9d462fbe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:02:10.869102 8871 sys_catalog.cc:455] T 00000000000000000000000000000000 P c30d606c6a834429bef8d6fe9d462fbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader c30d606c6a834429bef8d6fe9d462fbe. Latest consensus state: current_term: 1 leader_uuid: "c30d606c6a834429bef8d6fe9d462fbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30d606c6a834429bef8d6fe9d462fbe" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 35707 } } }
I20260925 03:02:10.869747 8871 sys_catalog.cc:458] T 00000000000000000000000000000000 P c30d606c6a834429bef8d6fe9d462fbe [sys.catalog]: This master's current role is: LEADER
I20260925 03:02:10.869130 8870 sys_catalog.cc:455] T 00000000000000000000000000000000 P c30d606c6a834429bef8d6fe9d462fbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c30d606c6a834429bef8d6fe9d462fbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30d606c6a834429bef8d6fe9d462fbe" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 35707 } } }
I20260925 03:02:10.870150 8870 sys_catalog.cc:458] T 00000000000000000000000000000000 P c30d606c6a834429bef8d6fe9d462fbe [sys.catalog]: This master's current role is: LEADER
I20260925 03:02:10.873291 8875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:02:10.878726 8875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:02:10.887341 8875 catalog_manager.cc:1347] Generated new cluster ID: 6f5b47bc1ac34a8f9115cbcf9ef5086f
I20260925 03:02:10.887449 8875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:02:10.914075 8875 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:02:10.914448 8875 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:02:10.920061 8875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c30d606c6a834429bef8d6fe9d462fbe: Generated new TSK 0
I20260925 03:02:10.920707 8875 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:02:10.924748 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 8792
W20260925 03:02:10.925702 8854 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52907
W20260925 03:02:10.926313 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 8792: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 03:02:10.926452 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 8792
2026-09-25T03:02:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1842 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 03:02:11 dist-test-slave-8zvk krb5kdc[8890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 03:02:11 dist-test-slave-8zvk krb5kdc[8890](info): set up 2 sockets
Sep 25 03:02:11 dist-test-slave-8zvk krb5kdc[8890](info): commencing operation
krb5kdc: starting...
W20260925 03:02:12.002602 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 03:02:12 dist-test-slave-8zvk krb5kdc[8890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T03:02:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:02:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 03:02:12.227667 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:44833
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:43839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:44833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:02:12.543690 8903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:02:12.544050 8903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:02:12.544224 8903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:02:12.551443 8903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:02:12.551604 8903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:02:12.551713 8903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:02:12.551823 8903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:02:12.551899 8903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:02:12.561684 8903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:43839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:44833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:44833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:02:12.563052 8903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:02:12.564360 8903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:02:12.575868 8908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:12.575831 8911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:12.575812 8909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:02:12.576663 8903 server_base.cc:1034] running on GCE node
I20260925 03:02:12.577313 8903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:02:12.578624 8903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:02:12.579813 8903 hybrid_clock.cc:648] HybridClock initialized: now 1790305332579752 us; error 50 us; skew 500 ppm
I20260925 03:02:12.580116 8903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:02:12 dist-test-slave-8zvk krb5kdc[8890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305332, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:02:12.583047 8903 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 03:02:12.584375 8903 webserver.cc:460] Webserver started at http://127.18.196.254:38885/ using document root <none> and password file <none>
I20260925 03:02:12.584998 8903 fs_manager.cc:359] Metadata directory not provided
I20260925 03:02:12.585093 8903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:02:12.585330 8903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:02:12.588197 8903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "2fb50dded3424ef38bb58e9f4fb60e8e"
format_stamp: "Formatted at 2026-09-25 03:02:12 on dist-test-slave-8zvk"
server_key: "06c4b94b7da50381c503fd274696b4f1"
server_key_iv: "57d4e7f73203ff7dca33a2e00ded5be8"
server_key_version: "encryptionkey@0"
I20260925 03:02:12.588969 8903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "2fb50dded3424ef38bb58e9f4fb60e8e"
format_stamp: "Formatted at 2026-09-25 03:02:12 on dist-test-slave-8zvk"
server_key: "06c4b94b7da50381c503fd274696b4f1"
server_key_iv: "57d4e7f73203ff7dca33a2e00ded5be8"
server_key_version: "encryptionkey@0"
I20260925 03:02:12.593720 8903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 03:02:12.596899 8918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:12.598167 8903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 03:02:12.598313 8903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "2fb50dded3424ef38bb58e9f4fb60e8e"
format_stamp: "Formatted at 2026-09-25 03:02:12 on dist-test-slave-8zvk"
server_key: "06c4b94b7da50381c503fd274696b4f1"
server_key_iv: "57d4e7f73203ff7dca33a2e00ded5be8"
server_key_version: "encryptionkey@0"
I20260925 03:02:12.598465 8903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:02:12.612782 8903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:02:12.617059 8903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:02:12.617292 8903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:02:12.635262 8903 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:44833
I20260925 03:02:12.635293 8970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:44833 every 8 connection(s)
I20260925 03:02:12.636559 8903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:02:12.639608 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 8903
I20260925 03:02:12.639791 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:02:12.640323 19219 external_mini_cluster.cc:1428] Setting key 2cee9361578f29abef29d70d6cbc9edb
I20260925 03:02:12.640503 8971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:12.654947 8971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fb50dded3424ef38bb58e9f4fb60e8e: Bootstrap starting.
Sep 25 03:02:12 dist-test-slave-8zvk krb5kdc[8890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:02:12.657969 8971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fb50dded3424ef38bb58e9f4fb60e8e: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:12.659338 8971 log.cc:824] T 00000000000000000000000000000000 P 2fb50dded3424ef38bb58e9f4fb60e8e: Log is configured to *not* fsync() on all Append() calls
I20260925 03:02:12.662612 8971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fb50dded3424ef38bb58e9f4fb60e8e: No bootstrap required, opened a new log
I20260925 03:02:12.664705 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:12.642020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43919 (local address 127.18.196.254:44833)
0925 03:02:12.642773 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:12.642784 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:12.642822 (+ 38us) server_negotiation.cc:407] Connection header received
0925 03:02:12.645024 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:12.645044 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:12.645413 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:12.645726 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:12.647534 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:12.648797 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:12.650599 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:12.651378 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:12.651535 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:12.656419 (+ 4884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:12.656426 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:12.656446 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:12.656486 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:12.660187 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:12.661020 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:12.661028 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:12.661035 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:12.661238 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:12.661606 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:12.661610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:12.661613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:12.662323 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:12.662641 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:12.662965 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 03:02:12.663433 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":361,"threads_started":1}
I20260925 03:02:12.668409 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 03:02:12.671562 8971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fb50dded3424ef38bb58e9f4fb60e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb50dded3424ef38bb58e9f4fb60e8e" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 44833 } }
I20260925 03:02:12.671919 8971 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fb50dded3424ef38bb58e9f4fb60e8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:12.672019 8971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fb50dded3424ef38bb58e9f4fb60e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb50dded3424ef38bb58e9f4fb60e8e, State: Initialized, Role: FOLLOWER
I20260925 03:02:12.672688 8971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fb50dded3424ef38bb58e9f4fb60e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb50dded3424ef38bb58e9f4fb60e8e" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 44833 } }
I20260925 03:02:12.672947 8971 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fb50dded3424ef38bb58e9f4fb60e8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:02:12.673123 8971 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fb50dded3424ef38bb58e9f4fb60e8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:02:12.673316 8971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fb50dded3424ef38bb58e9f4fb60e8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:12.676023 8971 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fb50dded3424ef38bb58e9f4fb60e8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb50dded3424ef38bb58e9f4fb60e8e" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 44833 } }
I20260925 03:02:12.676579 8971 leader_election.cc:302] T 00000000000000000000000000000000 P 2fb50dded3424ef38bb58e9f4fb60e8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fb50dded3424ef38bb58e9f4fb60e8e; no voters:
I20260925 03:02:12.677232 8971 leader_election.cc:288] T 00000000000000000000000000000000 P 2fb50dded3424ef38bb58e9f4fb60e8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:02:12.677345 8981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fb50dded3424ef38bb58e9f4fb60e8e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:02:12.679847 8971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fb50dded3424ef38bb58e9f4fb60e8e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:02:12.682446 8981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fb50dded3424ef38bb58e9f4fb60e8e [term 1 LEADER]: Becoming Leader. State: Replica: 2fb50dded3424ef38bb58e9f4fb60e8e, State: Running, Role: LEADER
I20260925 03:02:12.683346 8981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fb50dded3424ef38bb58e9f4fb60e8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb50dded3424ef38bb58e9f4fb60e8e" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 44833 } }
I20260925 03:02:12.687783 8983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fb50dded3424ef38bb58e9f4fb60e8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fb50dded3424ef38bb58e9f4fb60e8e. Latest consensus state: current_term: 1 leader_uuid: "2fb50dded3424ef38bb58e9f4fb60e8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb50dded3424ef38bb58e9f4fb60e8e" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 44833 } } }
I20260925 03:02:12.688110 8983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fb50dded3424ef38bb58e9f4fb60e8e [sys.catalog]: This master's current role is: LEADER
I20260925 03:02:12.688417 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:12.672891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56775 (local address 127.18.196.254:44833)
0925 03:02:12.673086 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:12.673094 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:12.673534 (+ 440us) server_negotiation.cc:407] Connection header received
0925 03:02:12.673682 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:12.673690 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:12.673774 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:12.673879 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:12.675297 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:12.676340 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:12.677863 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:12.678484 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:12.678558 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:12.681394 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:12.681403 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:12.681408 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:12.681468 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:12.684863 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:12.685749 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:12.685755 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:12.685759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:12.685923 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:12.686427 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:12.686444 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:12.686448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:12.687065 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:12.687458 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:12.687782 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 03:02:12.688033 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 03:02:12.687340 8982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fb50dded3424ef38bb58e9f4fb60e8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fb50dded3424ef38bb58e9f4fb60e8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb50dded3424ef38bb58e9f4fb60e8e" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 44833 } } }
I20260925 03:02:12.688659 8982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fb50dded3424ef38bb58e9f4fb60e8e [sys.catalog]: This master's current role is: LEADER
I20260925 03:02:12.689150 8991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:02:12.698915 8991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:02:12.708581 8991 catalog_manager.cc:1347] Generated new cluster ID: 4404e7d3e2e04813b36c69cfabfbc0df
I20260925 03:02:12.708716 8991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:02:12.727689 8991 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:02:12.728943 8991 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:02:12.737516 8991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fb50dded3424ef38bb58e9f4fb60e8e: Generated new TSK 0
I20260925 03:02:12.738289 8991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:02:12.743243 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 8903
I20260925 03:02:12.744354 8966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:02:12.835016 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 8903
2026-09-25T03:02:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1909 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 03:02:12 dist-test-slave-8zvk krb5kdc[9003](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 03:02:12 dist-test-slave-8zvk krb5kdc[9003](info): set up 2 sockets
Sep 25 03:02:12 dist-test-slave-8zvk krb5kdc[9003](info): commencing operation
krb5kdc: starting...
W20260925 03:02:13.918583 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 03:02:14 dist-test-slave-8zvk krb5kdc[9003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T03:02:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:02:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 03:02:14.161703 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:42733
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:44387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:42733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:02:14.478762 9016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:02:14.479094 9016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:02:14.479188 9016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:02:14.486258 9016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:02:14.486358 9016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:02:14.486404 9016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:02:14.486464 9016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:02:14.486507 9016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:02:14.496308 9016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:44387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:42733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:42733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:02:14.497475 9016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:02:14.498734 9016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:02:14.510692 9021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:14.510656 9024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:14.510632 9022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:02:14.511631 9016 server_base.cc:1034] running on GCE node
I20260925 03:02:14.512485 9016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:02:14.513859 9016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:02:14.515093 9016 hybrid_clock.cc:648] HybridClock initialized: now 1790305334515003 us; error 82 us; skew 500 ppm
I20260925 03:02:14.515419 9016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:02:14 dist-test-slave-8zvk krb5kdc[9003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305334, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:02:14.518409 9016 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 03:02:14.519814 9016 webserver.cc:460] Webserver started at http://127.18.196.254:39731/ using document root <none> and password file <none>
I20260925 03:02:14.520485 9016 fs_manager.cc:359] Metadata directory not provided
I20260925 03:02:14.520637 9016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:02:14.520936 9016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:02:14.523886 9016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "edc83ef5892b4dc08698c0f24d709f80"
format_stamp: "Formatted at 2026-09-25 03:02:14 on dist-test-slave-8zvk"
server_key: "7c8ac3ca25627c8bc291e3b01d487a18"
server_key_iv: "5c0d93637eb57e6b01d388ab88ff0103"
server_key_version: "encryptionkey@0"
I20260925 03:02:14.524724 9016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "edc83ef5892b4dc08698c0f24d709f80"
format_stamp: "Formatted at 2026-09-25 03:02:14 on dist-test-slave-8zvk"
server_key: "7c8ac3ca25627c8bc291e3b01d487a18"
server_key_iv: "5c0d93637eb57e6b01d388ab88ff0103"
server_key_version: "encryptionkey@0"
I20260925 03:02:14.529745 9016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 03:02:14.533012 9031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:14.534296 9016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 03:02:14.534476 9016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "edc83ef5892b4dc08698c0f24d709f80"
format_stamp: "Formatted at 2026-09-25 03:02:14 on dist-test-slave-8zvk"
server_key: "7c8ac3ca25627c8bc291e3b01d487a18"
server_key_iv: "5c0d93637eb57e6b01d388ab88ff0103"
server_key_version: "encryptionkey@0"
I20260925 03:02:14.534669 9016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:02:14.550634 9016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:02:14.554428 9016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:02:14.554735 9016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:02:14.572954 9016 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:42733
I20260925 03:02:14.572981 9082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:42733 every 8 connection(s)
I20260925 03:02:14.574270 9016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:02:14.575084 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 9016
I20260925 03:02:14.575273 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:02:14.575907 19219 external_mini_cluster.cc:1428] Setting key 56a0e9e00f4856a1e8bbc99a37625032
I20260925 03:02:14.578424 9083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:14.588754 9086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:14.577780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38201 (local address 127.18.196.254:42733)
0925 03:02:14.578614 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:14.578625 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:14.578666 (+ 41us) server_negotiation.cc:407] Connection header received
0925 03:02:14.580287 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:14.580308 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:14.580686 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:14.581120 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:14.581662 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:14.583087 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:14.584407 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:14.585066 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:14.585267 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:14.586477 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:14.586484 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:14.586497 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:02:14.587128 (+ 631us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:14.587203 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:02:14.587232 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:14.587357 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:14.587588 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 03:02:14.587849 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":381,"threads_started":1}
I20260925 03:02:14.588900 9083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edc83ef5892b4dc08698c0f24d709f80: Bootstrap starting.
I20260925 03:02:14.592176 9083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edc83ef5892b4dc08698c0f24d709f80: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:14.592365 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 03:02:14.593240 9083 log.cc:824] T 00000000000000000000000000000000 P edc83ef5892b4dc08698c0f24d709f80: Log is configured to *not* fsync() on all Append() calls
I20260925 03:02:14.596145 9083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edc83ef5892b4dc08698c0f24d709f80: No bootstrap required, opened a new log
I20260925 03:02:14.603917 9086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:14.597030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51359 (local address 127.18.196.254:42733)
0925 03:02:14.597218 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:14.597224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:14.597783 (+ 559us) server_negotiation.cc:407] Connection header received
0925 03:02:14.597958 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:14.597965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:14.598048 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:14.598162 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:14.598687 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:14.599862 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:14.601201 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:14.601937 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:14.602070 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:14.603160 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:14.603167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:14.603171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:02:14.603192 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:14.603237 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:02:14.603258 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:14.603360 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:14.603475 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 03:02:14.603577 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 03:02:14.605957 9083 raft_consensus.cc:348] T 00000000000000000000000000000000 P edc83ef5892b4dc08698c0f24d709f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc83ef5892b4dc08698c0f24d709f80" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 42733 } }
I20260925 03:02:14.606330 9083 raft_consensus.cc:374] T 00000000000000000000000000000000 P edc83ef5892b4dc08698c0f24d709f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:14.606442 9083 raft_consensus.cc:729] T 00000000000000000000000000000000 P edc83ef5892b4dc08698c0f24d709f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc83ef5892b4dc08698c0f24d709f80, State: Initialized, Role: FOLLOWER
I20260925 03:02:14.607096 9083 consensus_queue.cc:260] T 00000000000000000000000000000000 P edc83ef5892b4dc08698c0f24d709f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc83ef5892b4dc08698c0f24d709f80" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 42733 } }
I20260925 03:02:14.607402 9083 raft_consensus.cc:388] T 00000000000000000000000000000000 P edc83ef5892b4dc08698c0f24d709f80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:02:14.607565 9083 raft_consensus.cc:482] T 00000000000000000000000000000000 P edc83ef5892b4dc08698c0f24d709f80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:02:14.607746 9083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edc83ef5892b4dc08698c0f24d709f80 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:14.609917 9083 raft_consensus.cc:504] T 00000000000000000000000000000000 P edc83ef5892b4dc08698c0f24d709f80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc83ef5892b4dc08698c0f24d709f80" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 42733 } }
I20260925 03:02:14.610430 9083 leader_election.cc:302] T 00000000000000000000000000000000 P edc83ef5892b4dc08698c0f24d709f80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edc83ef5892b4dc08698c0f24d709f80; no voters:
I20260925 03:02:14.611094 9083 leader_election.cc:288] T 00000000000000000000000000000000 P edc83ef5892b4dc08698c0f24d709f80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:02:14.611245 9093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edc83ef5892b4dc08698c0f24d709f80 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:02:14.612259 9093 raft_consensus.cc:686] T 00000000000000000000000000000000 P edc83ef5892b4dc08698c0f24d709f80 [term 1 LEADER]: Becoming Leader. State: Replica: edc83ef5892b4dc08698c0f24d709f80, State: Running, Role: LEADER
I20260925 03:02:14.612829 9093 consensus_queue.cc:237] T 00000000000000000000000000000000 P edc83ef5892b4dc08698c0f24d709f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc83ef5892b4dc08698c0f24d709f80" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 42733 } }
I20260925 03:02:14.613384 9083 sys_catalog.cc:564] T 00000000000000000000000000000000 P edc83ef5892b4dc08698c0f24d709f80 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:02:14.615566 9094 sys_catalog.cc:455] T 00000000000000000000000000000000 P edc83ef5892b4dc08698c0f24d709f80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edc83ef5892b4dc08698c0f24d709f80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc83ef5892b4dc08698c0f24d709f80" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 42733 } } }
I20260925 03:02:14.615532 9095 sys_catalog.cc:455] T 00000000000000000000000000000000 P edc83ef5892b4dc08698c0f24d709f80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edc83ef5892b4dc08698c0f24d709f80. Latest consensus state: current_term: 1 leader_uuid: "edc83ef5892b4dc08698c0f24d709f80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc83ef5892b4dc08698c0f24d709f80" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 42733 } } }
I20260925 03:02:14.615985 9094 sys_catalog.cc:458] T 00000000000000000000000000000000 P edc83ef5892b4dc08698c0f24d709f80 [sys.catalog]: This master's current role is: LEADER
I20260925 03:02:14.616036 9095 sys_catalog.cc:458] T 00000000000000000000000000000000 P edc83ef5892b4dc08698c0f24d709f80 [sys.catalog]: This master's current role is: LEADER
I20260925 03:02:14.617565 9098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:02:14.625272 9098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:02:14.633881 9098 catalog_manager.cc:1347] Generated new cluster ID: f7b3bd56f687483da44b1f5f261b2759
I20260925 03:02:14.633993 9098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:02:14.646440 9098 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:02:14.647886 9098 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:02:14.660674 9098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edc83ef5892b4dc08698c0f24d709f80: Generated new TSK 0
I20260925 03:02:14.661471 9098 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:02:14.672326 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 9016
W20260925 03:02:14.673772 9078 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38201
W20260925 03:02:14.674571 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 9016: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 03:02:14.674710 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 9016
2026-09-25T03:02:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1835 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 03:02:14 dist-test-slave-8zvk krb5kdc[9114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 03:02:14 dist-test-slave-8zvk krb5kdc[9114](info): set up 2 sockets
Sep 25 03:02:14 dist-test-slave-8zvk krb5kdc[9114](info): commencing operation
krb5kdc: starting...
W20260925 03:02:15.770468 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 03:02:15 dist-test-slave-8zvk krb5kdc[9114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T03:02:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:02:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 03:02:16.020339 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:40553
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:45395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:40553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:02:16.339551 9127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:02:16.339876 9127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:02:16.340032 9127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:02:16.347244 9127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:02:16.347406 9127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:02:16.347507 9127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:02:16.347616 9127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:02:16.347692 9127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:02:16.357324 9127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:45395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:40553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:40553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:02:16.358577 9127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:02:16.359917 9127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:02:16.371371 9135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:16.371446 9133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:16.371470 9132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:02:16.372638 9127 server_base.cc:1034] running on GCE node
I20260925 03:02:16.373559 9127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:02:16.375012 9127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:02:16.376255 9127 hybrid_clock.cc:648] HybridClock initialized: now 1790305336376193 us; error 52 us; skew 500 ppm
I20260925 03:02:16.376629 9127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:02:16 dist-test-slave-8zvk krb5kdc[9114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305336, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:02:16.383659 9127 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 03:02:16.385130 9127 webserver.cc:460] Webserver started at http://127.18.196.254:32927/ using document root <none> and password file <none>
I20260925 03:02:16.385803 9127 fs_manager.cc:359] Metadata directory not provided
I20260925 03:02:16.385962 9127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:02:16.386264 9127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:02:16.389261 9127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "4cc895cf8812445d9fbd52d43a10938e"
format_stamp: "Formatted at 2026-09-25 03:02:16 on dist-test-slave-8zvk"
server_key: "938b900d9feca065816740a32bac1f91"
server_key_iv: "fb6adbe9069f9b5539d100213fce39dd"
server_key_version: "encryptionkey@0"
I20260925 03:02:16.390022 9127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "4cc895cf8812445d9fbd52d43a10938e"
format_stamp: "Formatted at 2026-09-25 03:02:16 on dist-test-slave-8zvk"
server_key: "938b900d9feca065816740a32bac1f91"
server_key_iv: "fb6adbe9069f9b5539d100213fce39dd"
server_key_version: "encryptionkey@0"
I20260925 03:02:16.395133 9127 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 03:02:16.398558 9142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:16.399803 9127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 03:02:16.399999 9127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "4cc895cf8812445d9fbd52d43a10938e"
format_stamp: "Formatted at 2026-09-25 03:02:16 on dist-test-slave-8zvk"
server_key: "938b900d9feca065816740a32bac1f91"
server_key_iv: "fb6adbe9069f9b5539d100213fce39dd"
server_key_version: "encryptionkey@0"
I20260925 03:02:16.400198 9127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:02:16.405738 9127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:02:16.410100 9127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:02:16.410398 9127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:02:16.428978 9127 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:40553
I20260925 03:02:16.429018 9193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:40553 every 8 connection(s)
I20260925 03:02:16.430402 9127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:02:16.434003 9194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:16.434598 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 9127
I20260925 03:02:16.434778 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:02:16.435252 19219 external_mini_cluster.cc:1428] Setting key b9a1ba27b5c68a4fab4d6a89018635bb
I20260925 03:02:16.444504 9197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:16.437079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39043 (local address 127.18.196.254:40553)
0925 03:02:16.437918 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:16.437938 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:16.437988 (+ 50us) server_negotiation.cc:407] Connection header received
0925 03:02:16.440752 (+ 2764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:16.440781 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:16.441236 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:16.441685 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:16.442636 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:16.442644 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:16.442661 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:02:16.442676 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:16.442738 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:02:16.442766 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:16.443003 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:16.443132 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 03:02:16.443310 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":360,"threads_started":1}
I20260925 03:02:16.448148 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 03:02:16.448187 9194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cc895cf8812445d9fbd52d43a10938e: Bootstrap starting.
I20260925 03:02:16.451975 9194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cc895cf8812445d9fbd52d43a10938e: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:16.453357 9194 log.cc:824] T 00000000000000000000000000000000 P 4cc895cf8812445d9fbd52d43a10938e: Log is configured to *not* fsync() on all Append() calls
I20260925 03:02:16.456054 9197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:16.453494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42169 (local address 127.18.196.254:40553)
0925 03:02:16.453660 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:16.453665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:16.454114 (+ 449us) server_negotiation.cc:407] Connection header received
0925 03:02:16.454256 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:16.454261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:16.454323 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:16.454452 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:16.455377 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:16.455383 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:16.455388 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 03:02:16.455406 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:16.455456 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 03:02:16.455475 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:16.455592 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:16.455719 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 03:02:16.455780 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 03:02:16.456456 9194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cc895cf8812445d9fbd52d43a10938e: No bootstrap required, opened a new log
I20260925 03:02:16.464210 9194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cc895cf8812445d9fbd52d43a10938e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc895cf8812445d9fbd52d43a10938e" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40553 } }
I20260925 03:02:16.464591 9194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cc895cf8812445d9fbd52d43a10938e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 03:02:16.464697 9194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cc895cf8812445d9fbd52d43a10938e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc895cf8812445d9fbd52d43a10938e, State: Initialized, Role: FOLLOWER
I20260925 03:02:16.465313 9194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cc895cf8812445d9fbd52d43a10938e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc895cf8812445d9fbd52d43a10938e" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40553 } }
I20260925 03:02:16.465581 9194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cc895cf8812445d9fbd52d43a10938e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 03:02:16.465726 9194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cc895cf8812445d9fbd52d43a10938e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 03:02:16.465907 9194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cc895cf8812445d9fbd52d43a10938e [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:16.468072 9194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cc895cf8812445d9fbd52d43a10938e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc895cf8812445d9fbd52d43a10938e" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40553 } }
I20260925 03:02:16.468482 9194 leader_election.cc:302] T 00000000000000000000000000000000 P 4cc895cf8812445d9fbd52d43a10938e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cc895cf8812445d9fbd52d43a10938e; no voters:
I20260925 03:02:16.469120 9194 leader_election.cc:288] T 00000000000000000000000000000000 P 4cc895cf8812445d9fbd52d43a10938e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 03:02:16.469234 9204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cc895cf8812445d9fbd52d43a10938e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:02:16.470211 9204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cc895cf8812445d9fbd52d43a10938e [term 1 LEADER]: Becoming Leader. State: Replica: 4cc895cf8812445d9fbd52d43a10938e, State: Running, Role: LEADER
I20260925 03:02:16.470773 9204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cc895cf8812445d9fbd52d43a10938e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc895cf8812445d9fbd52d43a10938e" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40553 } }
I20260925 03:02:16.471275 9194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cc895cf8812445d9fbd52d43a10938e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:02:16.473500 9205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cc895cf8812445d9fbd52d43a10938e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cc895cf8812445d9fbd52d43a10938e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc895cf8812445d9fbd52d43a10938e" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40553 } } }
I20260925 03:02:16.473932 9205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cc895cf8812445d9fbd52d43a10938e [sys.catalog]: This master's current role is: LEADER
I20260925 03:02:16.474364 9206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cc895cf8812445d9fbd52d43a10938e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cc895cf8812445d9fbd52d43a10938e. Latest consensus state: current_term: 1 leader_uuid: "4cc895cf8812445d9fbd52d43a10938e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc895cf8812445d9fbd52d43a10938e" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 40553 } } }
I20260925 03:02:16.474639 9206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cc895cf8812445d9fbd52d43a10938e [sys.catalog]: This master's current role is: LEADER
I20260925 03:02:16.476766 9213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:02:16.480912 9213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:02:16.488339 9213 catalog_manager.cc:1347] Generated new cluster ID: 484ba91df3724555af7f425ef7bad340
I20260925 03:02:16.488437 9213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:02:16.515627 9213 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:02:16.516094 9213 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:02:16.522934 9213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cc895cf8812445d9fbd52d43a10938e: Generated new TSK 0
I20260925 03:02:16.523712 9213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:02:16.532747 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 9127
W20260925 03:02:16.534283 9189 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39043
W20260925 03:02:16.535007 19219 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 9127: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 03:02:16.535166 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 9127
2026-09-25T03:02:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1863 ms)
[----------] 18 tests from AuthTokenIssuingTest (34337 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 03:02:16 dist-test-slave-8zvk krb5kdc[9225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 03:02:16 dist-test-slave-8zvk krb5kdc[9225](info): set up 2 sockets
Sep 25 03:02:16 dist-test-slave-8zvk krb5kdc[9225](info): commencing operation
krb5kdc: starting...
W20260925 03:02:17.622586 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 03:02:17 dist-test-slave-8zvk krb5kdc[9225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T03:02:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:02:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 03:02:17.858325 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:39067
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:39067,127.18.196.253:34079,127.18.196.252:37443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:02:18.181231 9238 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 03:02:18.181512 9238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:02:18.181582 9238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:02:18.181659 9238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:02:18.188784 9238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:02:18.188865 9238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:02:18.188928 9238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:02:18.188978 9238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:02:18.189033 9238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:02:18.198369 9238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:39067,127.18.196.253:34079,127.18.196.252:37443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:39067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:02:18.199522 9238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:02:18.200817 9238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:02:18.212263 9244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:18.212432 9243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:18.212589 9246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:02:18.213469 9238 server_base.cc:1034] running on GCE node
I20260925 03:02:18.214255 9238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:02:18.215504 9238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:02:18.216741 9238 hybrid_clock.cc:648] HybridClock initialized: now 1790305338216692 us; error 42 us; skew 500 ppm
I20260925 03:02:18.217054 9238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:02:18 dist-test-slave-8zvk krb5kdc[9225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305338, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:02:18.220093 9238 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 03:02:18.221524 9238 webserver.cc:460] Webserver started at http://127.18.196.254:39497/ using document root <none> and password file <none>
I20260925 03:02:18.222153 9238 fs_manager.cc:359] Metadata directory not provided
I20260925 03:02:18.222239 9238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:02:18.222498 9238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:02:18.225195 9238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "6a9f63a6483a4a098f826dec94371591"
format_stamp: "Formatted at 2026-09-25 03:02:18 on dist-test-slave-8zvk"
server_key: "0a5e04bf36e8e56eed52a2a379e16513"
server_key_iv: "30c60c7c6e13debdd8a711a5d4a9fe67"
server_key_version: "encryptionkey@0"
I20260925 03:02:18.225879 9238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "6a9f63a6483a4a098f826dec94371591"
format_stamp: "Formatted at 2026-09-25 03:02:18 on dist-test-slave-8zvk"
server_key: "0a5e04bf36e8e56eed52a2a379e16513"
server_key_iv: "30c60c7c6e13debdd8a711a5d4a9fe67"
server_key_version: "encryptionkey@0"
I20260925 03:02:18.230594 9238 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260925 03:02:18.233774 9253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:18.235057 9238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 03:02:18.235181 9238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "6a9f63a6483a4a098f826dec94371591"
format_stamp: "Formatted at 2026-09-25 03:02:18 on dist-test-slave-8zvk"
server_key: "0a5e04bf36e8e56eed52a2a379e16513"
server_key_iv: "30c60c7c6e13debdd8a711a5d4a9fe67"
server_key_version: "encryptionkey@0"
I20260925 03:02:18.235344 9238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:02:18.245136 9238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:02:18.248816 9238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:02:18.249029 9238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:02:18.267331 9238 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:39067
I20260925 03:02:18.267352 9305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:39067 every 8 connection(s)
I20260925 03:02:18.268886 9238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:02:18.271174 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 9238
I20260925 03:02:18.271395 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:02:18.272056 19219 external_mini_cluster.cc:1428] Setting key 20742e951cc2cf44c778888953cb4f39
I20260925 03:02:18.272675 9306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:18.279076 9306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39067 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 03:02:18 dist-test-slave-8zvk krb5kdc[9225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305338, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.196.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.253@KRBTEST.COM" created.
I20260925 03:02:18.300209 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.282081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55472 (local address 127.18.196.254:39067)
0925 03:02:18.283735 (+ 1654us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:18.283743 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:18.283789 (+ 46us) server_negotiation.cc:407] Connection header received
0925 03:02:18.285277 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:18.285282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:18.285493 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:18.285697 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:18.286928 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:18.288079 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:18.289445 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:18.289989 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:18.290183 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:18.293512 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:18.293519 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:18.293531 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:18.293558 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:18.296374 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:18.297085 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:18.297095 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:18.297099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:18.297260 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:18.297662 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:18.297666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:18.297668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:18.298121 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:18.298271 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:18.298988 (+ 717us) server_negotiation.cc:299] Negotiation successful
0925 03:02:18.299217 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1440,"thread_start_us":1233,"threads_started":1}
I20260925 03:02:18.300243 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.282080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39067 (local address 127.0.0.1:55472)
0925 03:02:18.282940 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:18.282985 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 03:02:18.284914 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:02:18.285699 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:02:18.285707 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:02:18.286383 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:02:18.286695 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:18.286712 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:18.288248 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:18.288252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:18.289295 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:18.289308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:18.290225 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:18.290232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:18.290404 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:18.291075 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:02:18.291095 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:02:18.293291 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:02:18.296516 (+ 3225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:18.296532 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:18.296542 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:18.296955 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:18.297365 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:18.297371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:18.297376 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:18.297525 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:18.298275 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:02:18.298285 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:02:18.298523 (+ 238us) client_negotiation.cc:769] Sending connection context
0925 03:02:18.298817 (+ 294us) client_negotiation.cc:240] Negotiation successful
0925 03:02:18.299428 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":66,"thread_start_us":372,"threads_started":1}
I20260925 03:02:18.303102 9306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } has no permanent_uuid. Determining permanent_uuid...
I20260925 03:02:18.304220 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.303858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:34079 (local address 127.0.0.1:47274)
0925 03:02:18.304029 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:18.304125 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:34079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260925 03:02:18.304489 9257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.196.253:34079: connect: Connection refused (error 111)
I20260925 03:02:18.305967 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.305624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:34079 (local address 127.0.0.1:47290)
0925 03:02:18.305756 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:18.305825 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:34079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 03:02:18.306177 9306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.253:34079: Network error: Client connection negotiation failed: client connection to 127.18.196.253:34079: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.253.keytab.
Entry for principal kudu/127.18.196.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.253.keytab.
I20260925 03:02:18.327858 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.253:34079
--webserver_interface=127.18.196.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.253.keytab
--principal=kudu/127.18.196.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:39067,127.18.196.253:34079,127.18.196.252:37443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 03:02:18.355469 9306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } attempt: 1
I20260925 03:02:18.356657 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.356145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:34079 (local address 127.0.0.1:47302)
0925 03:02:18.356356 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:18.356498 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:34079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 03:02:18.357605 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.357246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:34079 (local address 127.0.0.1:47318)
0925 03:02:18.357383 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:18.357461 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:34079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 03:02:18.357774 9306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.253:34079: Network error: Client connection negotiation failed: client connection to 127.18.196.253:34079: connect: Connection refused (error 111)
I20260925 03:02:18.426023 9306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } attempt: 2
I20260925 03:02:18.427307 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.426721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:34079 (local address 127.0.0.1:47326)
0925 03:02:18.427023 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:18.427134 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:34079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 03:02:18.428463 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.428050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:34079 (local address 127.0.0.1:47330)
0925 03:02:18.428226 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:18.428311 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:34079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 03:02:18.428709 9306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.253:34079: Network error: Client connection negotiation failed: client connection to 127.18.196.253:34079: connect: Connection refused (error 111)
I20260925 03:02:18.519981 9306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } attempt: 3
I20260925 03:02:18.521207 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.520653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:34079 (local address 127.0.0.1:47338)
0925 03:02:18.520887 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:18.521014 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:34079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 03:02:18.522289 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.521872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:34079 (local address 127.0.0.1:47344)
0925 03:02:18.522036 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:18.522121 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:34079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 03:02:18.522482 9306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.253:34079: Network error: Client connection negotiation failed: client connection to 127.18.196.253:34079: connect: Connection refused (error 111)
W20260925 03:02:18.654804 9313 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 03:02:18.655120 9313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:02:18.655190 9313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:02:18.655262 9313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:02:18.662233 9313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:02:18.662319 9313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:02:18.662375 9313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:02:18.662431 9313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:02:18.662493 9313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:02:18.665751 9306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } attempt: 4
I20260925 03:02:18.667022 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.666382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:34079 (local address 127.0.0.1:47354)
0925 03:02:18.666707 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:18.666813 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:34079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260925 03:02:18.667979 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.667649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:34079 (local address 127.0.0.1:47370)
0925 03:02:18.667776 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:18.667851 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:34079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 03:02:18.668242 9306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.253:34079: Network error: Client connection negotiation failed: client connection to 127.18.196.253:34079: connect: Connection refused (error 111)
I20260925 03:02:18.673502 9313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:39067,127.18.196.253:34079,127.18.196.252:37443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.253.keytab
--principal=kudu/127.18.196.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.253:34079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.196.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:02:18.674672 9313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:02:18.675952 9313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:02:18.687417 9318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:18.687568 9319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:18.687448 9321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:02:18.688472 9313 server_base.cc:1034] running on GCE node
I20260925 03:02:18.689199 9313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:02:18.690572 9313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:02:18.691808 9313 hybrid_clock.cc:648] HybridClock initialized: now 1790305338691726 us; error 61 us; skew 500 ppm
I20260925 03:02:18.692123 9313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:02:18 dist-test-slave-8zvk krb5kdc[9225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305338, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:02:18.695188 9313 init.cc:377] Logged in from keytab as kudu/127.18.196.253@KRBTEST.COM (short username kudu)
I20260925 03:02:18.696734 9313 webserver.cc:460] Webserver started at http://127.18.196.253:34701/ using document root <none> and password file <none>
I20260925 03:02:18.697417 9313 fs_manager.cc:359] Metadata directory not provided
I20260925 03:02:18.697551 9313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:02:18.697873 9313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:02:18.700827 9313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-1/data/instance:
uuid: "31132652ea494bcca45b0f9f6e90334e"
format_stamp: "Formatted at 2026-09-25 03:02:18 on dist-test-slave-8zvk"
server_key: "dfa8ea5d110f7500a966c72993e309c1"
server_key_iv: "482a576dae251a7475db0417518f9a95"
server_key_version: "encryptionkey@0"
I20260925 03:02:18.701622 9313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-1/wal/instance:
uuid: "31132652ea494bcca45b0f9f6e90334e"
format_stamp: "Formatted at 2026-09-25 03:02:18 on dist-test-slave-8zvk"
server_key: "dfa8ea5d110f7500a966c72993e309c1"
server_key_iv: "482a576dae251a7475db0417518f9a95"
server_key_version: "encryptionkey@0"
I20260925 03:02:18.706643 9313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 03:02:18.709962 9328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:18.711248 9313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 03:02:18.711426 9313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-1/wal
uuid: "31132652ea494bcca45b0f9f6e90334e"
format_stamp: "Formatted at 2026-09-25 03:02:18 on dist-test-slave-8zvk"
server_key: "dfa8ea5d110f7500a966c72993e309c1"
server_key_iv: "482a576dae251a7475db0417518f9a95"
server_key_version: "encryptionkey@0"
I20260925 03:02:18.711638 9313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:02:18.744679 9313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:02:18.748368 9313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:02:18.748631 9313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:02:18.766471 9313 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.253:34079
I20260925 03:02:18.766490 9379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.253:34079 every 8 connection(s)
I20260925 03:02:18.767772 9313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-1/data/info.pb
I20260925 03:02:18.771231 9380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:18.771237 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 9313
I20260925 03:02:18.771415 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-1/wal/instance
I20260925 03:02:18.771898 19219 external_mini_cluster.cc:1428] Setting key f582c0773b255f2a834ced03b9c923eb
I20260925 03:02:18.777207 9380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39067 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 03:02:18 dist-test-slave-8zvk krb5kdc[9225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305338, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.253@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.196.252@KRBTEST.COM; defaulting to no policy
I20260925 03:02:18.798111 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.779696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55478 (local address 127.18.196.254:39067)
0925 03:02:18.779924 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:18.779931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:18.781049 (+ 1118us) server_negotiation.cc:407] Connection header received
0925 03:02:18.782873 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:18.782881 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:18.782975 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:18.783091 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:18.784456 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:18.785547 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:18.787014 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:18.787754 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:18.787890 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:18.791352 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:18.791361 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:18.791368 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:18.791423 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:18.794993 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:18.795717 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:18.795725 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:18.795731 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:18.795890 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:18.796276 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:18.796284 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:18.796289 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:18.796679 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:18.796795 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:18.797529 (+ 734us) server_negotiation.cc:299] Negotiation successful
0925 03:02:18.797751 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.252@KRBTEST.COM" created.
I20260925 03:02:18.799142 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.780094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39067 (local address 127.0.0.1:55478)
0925 03:02:18.780838 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:18.780914 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0925 03:02:18.782589 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:02:18.783272 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:02:18.783278 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:02:18.783746 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:02:18.784279 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:18.784295 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:18.785692 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:18.785696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:18.786804 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:18.786815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:18.787974 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:18.787979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:18.788153 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:18.788828 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:02:18.788849 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:02:18.791134 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:02:18.795141 (+ 4007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:18.795147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:18.795158 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:18.795583 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:18.796001 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:18.796004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:18.796006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:18.796159 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:18.796791 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:02:18.796796 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:02:18.797094 (+ 298us) client_negotiation.cc:769] Sending connection context
0925 03:02:18.797428 (+ 334us) client_negotiation.cc:240] Negotiation successful
0925 03:02:18.797912 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":329,"threads_started":1}
I20260925 03:02:18.800765 9380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 03:02:18 dist-test-slave-8zvk krb5kdc[9225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305338, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.253@KRBTEST.COM for kudu/127.18.196.253@KRBTEST.COM
I20260925 03:02:18.816646 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.801538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:34079 (local address 127.0.0.1:47386)
0925 03:02:18.801743 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:18.801763 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 03:02:18.801883 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:02:18.802804 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:02:18.802812 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:02:18.803329 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:02:18.803529 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:18.803543 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:18.804833 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:18.804841 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:18.806286 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:18.806303 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:18.807164 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:18.807172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:18.807414 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:18.807973 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:02:18.807989 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:02:18.809831 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:02:18.814072 (+ 4241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:18.814075 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:18.814078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:18.814419 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:18.814925 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:18.814930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:18.814932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:18.815067 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:18.815854 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:02:18.815857 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:02:18.816012 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 03:02:18.816122 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 03:02:18.816360 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 03:02:18.817294 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.801739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47386 (local address 127.18.196.253:34079)
0925 03:02:18.802416 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:18.802422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:18.802445 (+ 23us) server_negotiation.cc:407] Connection header received
0925 03:02:18.802485 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:18.802490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:18.802641 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:18.802744 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:18.803691 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:18.804697 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:18.806422 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:18.807008 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:18.807099 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:18.810066 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:18.810081 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:18.810099 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:18.810139 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:18.813935 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:18.814548 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:18.814558 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:18.814564 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:18.814700 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:18.815205 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:18.815213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:18.815219 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:18.815718 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:18.815988 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:18.816339 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 03:02:18.816646 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":390,"threads_started":1}
I20260925 03:02:18.819651 9380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } has no permanent_uuid. Determining permanent_uuid...
I20260925 03:02:18.820669 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.820242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:37443 (local address 127.0.0.1:48406)
0925 03:02:18.820410 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:18.820535 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 03:02:18.820930 9332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
I20260925 03:02:18.822698 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.822215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:37443 (local address 127.0.0.1:48420)
0925 03:02:18.822430 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:18.822543 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 03:02:18.822943 9380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:37443: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.252.keytab.
Entry for principal kudu/127.18.196.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.252.keytab.
I20260925 03:02:18.833515 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.252:37443
--webserver_interface=127.18.196.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.252.keytab
--principal=kudu/127.18.196.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:39067,127.18.196.253:34079,127.18.196.252:37443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 03:02:18.872346 9380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } attempt: 1
I20260925 03:02:18.873594 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.873057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:37443 (local address 127.0.0.1:48430)
0925 03:02:18.873275 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:18.873409 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 03:02:18.874630 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.874234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:37443 (local address 127.0.0.1:48442)
0925 03:02:18.874413 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:18.874502 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 03:02:18.874894 9380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:37443: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
I20260925 03:02:18.943141 9380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } attempt: 2
I20260925 03:02:18.944355 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.943827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:37443 (local address 127.0.0.1:48448)
0925 03:02:18.944047 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:18.944155 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 03:02:18.945355 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.944990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:37443 (local address 127.0.0.1:48456)
0925 03:02:18.945134 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:18.945216 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 03:02:18.945564 9380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:37443: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
I20260925 03:02:18.967592 9306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } attempt: 5
Sep 25 03:02:18 dist-test-slave-8zvk krb5kdc[9225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305338, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for kudu/127.18.196.253@KRBTEST.COM
I20260925 03:02:18.981918 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.968322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:34079 (local address 127.0.0.1:47394)
0925 03:02:18.968513 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:18.968534 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 03:02:18.968640 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:02:18.969014 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:02:18.969017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:02:18.969429 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:02:18.969611 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:18.969620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:18.971028 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:18.971033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:18.971856 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:18.971867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:18.972936 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:18.972940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:18.973097 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:18.973710 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:02:18.973727 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:02:18.975870 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:02:18.979771 (+ 3901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:18.979774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:18.979777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:18.980135 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:18.980512 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:18.980515 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:18.980517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:18.980635 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:18.981193 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:02:18.981196 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:02:18.981323 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 03:02:18.981425 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 03:02:18.981631 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 03:02:18.982110 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.968414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47394 (local address 127.18.196.253:34079)
0925 03:02:18.968623 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:18.968629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:18.968652 (+ 23us) server_negotiation.cc:407] Connection header received
0925 03:02:18.968769 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:18.968775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:18.968870 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:18.968984 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:18.969773 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:18.970896 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:18.972022 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:18.972797 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:18.972925 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:18.976085 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:18.976094 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:18.976101 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:18.976146 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:18.979649 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:18.980249 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:18.980255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:18.980259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:18.980408 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:18.980735 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:18.980740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:18.980743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:18.981078 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:18.981218 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:18.981551 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 03:02:18.981760 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 03:02:18.983134 9306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } has no permanent_uuid. Determining permanent_uuid...
I20260925 03:02:18.984061 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.983690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:37443 (local address 127.0.0.1:48460)
0925 03:02:18.983856 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:18.983943 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 03:02:18.984920 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:18.984596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:37443 (local address 127.0.0.1:48466)
0925 03:02:18.984743 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:18.984828 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 03:02:18.985138 9306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:37443: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
I20260925 03:02:19.036350 9306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } attempt: 1
I20260925 03:02:19.036775 9380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } attempt: 3
I20260925 03:02:19.037653 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.037097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:37443 (local address 127.0.0.1:48482)
0925 03:02:19.037326 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:19.037461 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 03:02:19.037778 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.037341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:37443 (local address 127.0.0.1:48498)
0925 03:02:19.037523 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:19.037605 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 03:02:19.038898 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.038477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:37443 (local address 127.0.0.1:48512)
0925 03:02:19.038650 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:19.038738 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 03:02:19.039131 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.038543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:37443 (local address 127.0.0.1:48516)
0925 03:02:19.038799 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:19.038993 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260925 03:02:19.039328 9380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:37443: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
W20260925 03:02:19.039433 9306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:37443: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
I20260925 03:02:19.107692 9306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } attempt: 2
I20260925 03:02:19.108913 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.108337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:37443 (local address 127.0.0.1:48518)
0925 03:02:19.108573 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:19.108684 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 03:02:19.110033 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.109619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:37443 (local address 127.0.0.1:48530)
0925 03:02:19.109782 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:19.109875 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 03:02:19.110244 9306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:37443: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
W20260925 03:02:19.160646 9387 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 03:02:19.160917 9387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:02:19.160983 9387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:02:19.161055 9387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:02:19.167608 9387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:02:19.167673 9387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:02:19.167721 9387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:02:19.167764 9387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:02:19.167812 9387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:02:19.176703 9387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:39067,127.18.196.253:34079,127.18.196.252:37443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.252.keytab
--principal=kudu/127.18.196.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.252:37443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.196.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:02:19.177754 9387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:02:19.178925 9387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 03:02:19.182536 9380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } attempt: 4
I20260925 03:02:19.183801 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.183240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:37443 (local address 127.0.0.1:48534)
0925 03:02:19.183507 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:19.183620 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 03:02:19.184892 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.184478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:37443 (local address 127.0.0.1:48544)
0925 03:02:19.184627 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:19.184719 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 03:02:19.185138 9380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:37443: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
W20260925 03:02:19.190850 9392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:19.191180 9393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:19.191196 9395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:02:19.191928 9387 server_base.cc:1034] running on GCE node
I20260925 03:02:19.192597 9387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:02:19.193838 9387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:02:19.195072 9387 hybrid_clock.cc:648] HybridClock initialized: now 1790305339195003 us; error 48 us; skew 500 ppm
I20260925 03:02:19.195385 9387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:02:19 dist-test-slave-8zvk krb5kdc[9225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305339, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:02:19.198360 9387 init.cc:377] Logged in from keytab as kudu/127.18.196.252@KRBTEST.COM (short username kudu)
I20260925 03:02:19.199717 9387 webserver.cc:460] Webserver started at http://127.18.196.252:42867/ using document root <none> and password file <none>
I20260925 03:02:19.200362 9387 fs_manager.cc:359] Metadata directory not provided
I20260925 03:02:19.200500 9387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:02:19.200798 9387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:02:19.203635 9387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-2/data/instance:
uuid: "282c7db188154044a20fcafdb7fad72a"
format_stamp: "Formatted at 2026-09-25 03:02:19 on dist-test-slave-8zvk"
server_key: "e3042cb12fe08472b2feb6f9b9af6a56"
server_key_iv: "19f9b96de835e8bc8ff955aebe80abe2"
server_key_version: "encryptionkey@0"
I20260925 03:02:19.204368 9387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-2/wal/instance:
uuid: "282c7db188154044a20fcafdb7fad72a"
format_stamp: "Formatted at 2026-09-25 03:02:19 on dist-test-slave-8zvk"
server_key: "e3042cb12fe08472b2feb6f9b9af6a56"
server_key_iv: "19f9b96de835e8bc8ff955aebe80abe2"
server_key_version: "encryptionkey@0"
I20260925 03:02:19.209260 9387 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 03:02:19.212386 9402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:19.213603 9387 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 03:02:19.213774 9387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-2/wal
uuid: "282c7db188154044a20fcafdb7fad72a"
format_stamp: "Formatted at 2026-09-25 03:02:19 on dist-test-slave-8zvk"
server_key: "e3042cb12fe08472b2feb6f9b9af6a56"
server_key_iv: "19f9b96de835e8bc8ff955aebe80abe2"
server_key_version: "encryptionkey@0"
I20260925 03:02:19.213996 9387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:02:19.216482 9306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } attempt: 3
I20260925 03:02:19.217774 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.217048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:37443 (local address 127.0.0.1:48558)
0925 03:02:19.217387 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:19.217514 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 03:02:19.219115 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.218616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:37443 (local address 127.0.0.1:48572)
0925 03:02:19.218871 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:19.218960 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 03:02:19.219399 9306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:37443: Network error: Client connection negotiation failed: client connection to 127.18.196.252:37443: connect: Connection refused (error 111)
I20260925 03:02:19.250003 9387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:02:19.254221 9387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:02:19.254509 9387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:02:19.272231 9387 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.252:37443
I20260925 03:02:19.272246 9453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.252:37443 every 8 connection(s)
I20260925 03:02:19.273730 9387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-2/data/info.pb
I20260925 03:02:19.277285 9454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:19.277881 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 9387
I20260925 03:02:19.278079 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304987644417-19219-0/minicluster-data/master-2/wal/instance
I20260925 03:02:19.278527 19219 external_mini_cluster.cc:1428] Setting key c92e069b05caae5898d49cd39385407c
I20260925 03:02:19.286113 9454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39067 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 03:02:19 dist-test-slave-8zvk krb5kdc[9225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.252@KRBTEST.COM
Sep 25 03:02:19 dist-test-slave-8zvk krb5kdc[9225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305339, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.252@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:02:19.300426 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.280310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48582 (local address 127.18.196.252:37443)
0925 03:02:19.281100 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:19.281111 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:19.281151 (+ 40us) server_negotiation.cc:407] Connection header received
0925 03:02:19.283157 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:19.283177 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:19.283549 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:19.283858 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:19.285540 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.286775 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.288611 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.289389 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.289588 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:19.293761 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:19.293769 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:19.293779 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:19.293818 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:19.296798 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:19.297458 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:19.297465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:19.297470 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:19.297602 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:19.297995 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:19.298001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:19.298004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:19.298425 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:19.298714 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:19.299206 (+ 492us) server_negotiation.cc:299] Negotiation successful
0925 03:02:19.299553 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":355,"threads_started":1}
I20260925 03:02:19.302222 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.288711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55490 (local address 127.18.196.254:39067)
0925 03:02:19.288902 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:19.288908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:19.289557 (+ 649us) server_negotiation.cc:407] Connection header received
0925 03:02:19.289796 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:19.289803 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:19.289872 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:19.289995 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:19.291046 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.291701 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.293052 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.293488 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.293620 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:19.296836 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:19.296841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:19.296845 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:19.296875 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:19.299693 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:19.300347 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:19.300352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:19.300355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:19.300482 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:19.300864 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:19.300869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:19.300872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:19.301177 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:19.301363 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:19.301795 (+ 432us) server_negotiation.cc:299] Negotiation successful
0925 03:02:19.301989 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 03:02:19.302538 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.288642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:39067 (local address 127.0.0.1:55490)
0925 03:02:19.289439 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:19.289477 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 03:02:19.289673 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:02:19.290063 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:02:19.290069 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:02:19.290621 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:02:19.290898 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:19.290912 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.291844 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.291849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:19.292908 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:19.292922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.293722 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.293731 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:19.293922 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:19.294507 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:02:19.294522 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:02:19.296647 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:02:19.299821 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:19.299831 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:19.299837 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:19.300236 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:19.300618 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:19.300622 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:19.300625 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:19.300757 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:19.301331 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:02:19.301336 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:02:19.301486 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 03:02:19.301701 (+ 215us) client_negotiation.cc:240] Negotiation successful
0925 03:02:19.302036 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":432,"threads_started":1}
I20260925 03:02:19.303879 9454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 03:02:19 dist-test-slave-8zvk krb5kdc[9225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305339, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.252@KRBTEST.COM for kudu/127.18.196.253@KRBTEST.COM
Sep 25 03:02:19 dist-test-slave-8zvk krb5kdc[9225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:02:19.316645 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.304580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:34079 (local address 127.0.0.1:47398)
0925 03:02:19.304710 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:19.304727 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 03:02:19.304807 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:02:19.305099 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:02:19.305102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:02:19.305404 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:02:19.305522 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:19.305529 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.306431 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.306438 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:19.307260 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:19.307271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.307944 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.307950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:19.308121 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:19.308725 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:02:19.308742 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:02:19.310738 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:02:19.314321 (+ 3583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:19.314325 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:19.314327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:19.314647 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:19.315060 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:19.315066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:19.315072 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:19.315225 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:19.315787 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:02:19.315791 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:02:19.316057 (+ 266us) client_negotiation.cc:769] Sending connection context
0925 03:02:19.316202 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 03:02:19.316397 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 03:02:19.317019 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.304593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47398 (local address 127.18.196.253:34079)
0925 03:02:19.304783 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:19.304789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:19.304810 (+ 21us) server_negotiation.cc:407] Connection header received
0925 03:02:19.304897 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:19.304902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:19.304980 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:19.305070 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:19.305663 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.306301 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.307388 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.307811 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.307933 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:19.310985 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:19.310989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:19.310992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:19.311023 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:19.314183 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:19.314758 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:19.314766 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:19.314771 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:19.314953 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:19.315332 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:19.315335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:19.315337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:19.315629 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:19.315791 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:19.316312 (+ 521us) server_negotiation.cc:299] Negotiation successful
0925 03:02:19.316551 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 03:02:19.317862 9454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } has no permanent_uuid. Determining permanent_uuid...
I20260925 03:02:19.318166 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.303625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55504 (local address 127.18.196.254:39067)
0925 03:02:19.303762 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:19.303770 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:19.303821 (+ 51us) server_negotiation.cc:407] Connection header received
0925 03:02:19.303997 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:19.304002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:19.304066 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:19.304145 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:19.305467 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.306168 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.307318 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.307725 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.307847 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:19.311722 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:19.311730 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:19.311736 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:19.311774 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:19.314682 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:19.315563 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:19.315568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:19.315571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:19.315717 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:19.316233 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:19.316237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:19.316239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:19.316607 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:19.316856 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:19.317704 (+ 848us) server_negotiation.cc:299] Negotiation successful
0925 03:02:19.317933 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 03:02:19 dist-test-slave-8zvk krb5kdc[9225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305339, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.252@KRBTEST.COM for kudu/127.18.196.252@KRBTEST.COM
Sep 25 03:02:19 dist-test-slave-8zvk krb5kdc[9225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.253@KRBTEST.COM
I20260925 03:02:19.330631 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.318575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48590 (local address 127.18.196.252:37443)
0925 03:02:19.318714 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:19.318721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:19.318742 (+ 21us) server_negotiation.cc:407] Connection header received
0925 03:02:19.318818 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:19.318823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:19.318978 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:19.319095 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:19.319738 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.320626 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.322059 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.322673 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.322767 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:19.325638 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:19.325643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:19.325647 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:19.325681 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:19.328057 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:19.328629 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:19.328636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:19.328639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:19.328805 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:19.329243 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:19.329247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:19.329249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:19.329501 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:19.329601 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:19.330078 (+ 477us) server_negotiation.cc:299] Negotiation successful
0925 03:02:19.330398 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":130,"server-negotiator.queue_time_us":34}
I20260925 03:02:19.330650 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.318415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:37443 (local address 127.0.0.1:48590)
0925 03:02:19.318636 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:19.318650 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 03:02:19.318728 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:02:19.319109 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:02:19.319113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:02:19.319459 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:02:19.319572 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:19.319580 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.320749 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.320755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:19.321937 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:19.321948 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.322811 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.322817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:19.323068 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:19.323566 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:02:19.323584 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:02:19.325481 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:02:19.328184 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:19.328191 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:19.328194 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:19.328523 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:19.328940 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:19.328946 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:19.328953 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:19.329128 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:19.329622 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:02:19.329630 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:02:19.329796 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 03:02:19.329988 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 03:02:19.330268 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260925 03:02:19.333434 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.319850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47400 (local address 127.18.196.253:34079)
0925 03:02:19.320032 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:19.320037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:19.320052 (+ 15us) server_negotiation.cc:407] Connection header received
0925 03:02:19.320203 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:19.320208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:19.320267 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:19.320350 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:19.321637 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.322504 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.324115 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.324459 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.324585 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:19.328327 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:19.328331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:19.328334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:19.328359 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:19.331317 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:19.331940 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:19.331943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:19.331945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:19.332056 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:19.332425 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:19.332428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:19.332430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:19.332639 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:19.332815 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:19.333104 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 03:02:19.333237 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 03:02:19.335671 9454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 282c7db188154044a20fcafdb7fad72a: Bootstrap starting.
I20260925 03:02:19.339277 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 03:02:19.339459 9454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 282c7db188154044a20fcafdb7fad72a: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:19.340652 9454 log.cc:824] T 00000000000000000000000000000000 P 282c7db188154044a20fcafdb7fad72a: Log is configured to *not* fsync() on all Append() calls
I20260925 03:02:19.343302 9454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 282c7db188154044a20fcafdb7fad72a: No bootstrap required, opened a new log
I20260925 03:02:19.342561 9281 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6a9f63a6483a4a098f826dec94371591"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55504
I20260925 03:02:19.349723 9454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 282c7db188154044a20fcafdb7fad72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f63a6483a4a098f826dec94371591" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39067 } } peers { permanent_uuid: "31132652ea494bcca45b0f9f6e90334e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } } peers { permanent_uuid: "282c7db188154044a20fcafdb7fad72a" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } }
I20260925 03:02:19.350024 9454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 282c7db188154044a20fcafdb7fad72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282c7db188154044a20fcafdb7fad72a, State: Initialized, Role: FOLLOWER
I20260925 03:02:19.350455 9454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 282c7db188154044a20fcafdb7fad72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f63a6483a4a098f826dec94371591" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39067 } } peers { permanent_uuid: "31132652ea494bcca45b0f9f6e90334e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } } peers { permanent_uuid: "282c7db188154044a20fcafdb7fad72a" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } }
I20260925 03:02:19.351320 9460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282c7db188154044a20fcafdb7fad72a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f63a6483a4a098f826dec94371591" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39067 } } peers { permanent_uuid: "31132652ea494bcca45b0f9f6e90334e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } } peers { permanent_uuid: "282c7db188154044a20fcafdb7fad72a" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } } }
I20260925 03:02:19.351704 9460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282c7db188154044a20fcafdb7fad72a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 03:02:19.352181 9454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 282c7db188154044a20fcafdb7fad72a [sys.catalog]: configured and running, proceeding with master startup.
W20260925 03:02:19.360541 9471 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 282c7db188154044a20fcafdb7fad72a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 03:02:19.360697 9471 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 03:02:19.358681 9387 master_runner.cc:186] Error getting master registration for 127.18.196.254:39067: OK, instance_id { permanent_uuid: "6a9f63a6483a4a098f826dec94371591" instance_seqno: 1790305338262169 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 03:02:19.369547 9387 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 03:02:19.396615 9306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } attempt: 4
Sep 25 03:02:19 dist-test-slave-8zvk krb5kdc[9225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305338, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for kudu/127.18.196.252@KRBTEST.COM
I20260925 03:02:19.411764 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.397351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:37443 (local address 127.0.0.1:48592)
0925 03:02:19.397639 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:19.397657 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 03:02:19.397817 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:02:19.398182 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:02:19.398189 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:02:19.398598 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:02:19.398793 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:19.398803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.399960 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.399966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:19.400904 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:19.400916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.401643 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.401649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:19.401862 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:19.402476 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:02:19.402494 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:02:19.404798 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:02:19.408905 (+ 4107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:19.408911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:19.408915 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:19.409443 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:19.409884 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:19.409888 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:19.409892 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:19.410052 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:19.410745 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:02:19.410749 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:02:19.410947 (+ 198us) client_negotiation.cc:769] Sending connection context
0925 03:02:19.411094 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 03:02:19.411353 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260925 03:02:19.411821 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.397440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48592 (local address 127.18.196.252:37443)
0925 03:02:19.397659 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:19.397666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:19.397740 (+ 74us) server_negotiation.cc:407] Connection header received
0925 03:02:19.397966 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:19.397971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:19.398030 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:19.398144 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:19.399014 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.399784 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.401050 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.401477 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.401616 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:19.405000 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:19.405007 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:19.405014 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:19.405056 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:19.408726 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:19.409612 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:19.409619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:19.409622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:19.409762 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:19.410175 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:19.410181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:19.410185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:19.410623 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:19.410795 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:19.411219 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 03:02:19.411450 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 03:02:19.416767 9306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591: Bootstrap starting.
I20260925 03:02:19.419890 9306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:19.420920 9306 log.cc:824] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591: Log is configured to *not* fsync() on all Append() calls
I20260925 03:02:19.423734 9306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591: No bootstrap required, opened a new log
I20260925 03:02:19.428870 9306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f63a6483a4a098f826dec94371591" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39067 } } peers { permanent_uuid: "31132652ea494bcca45b0f9f6e90334e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } } peers { permanent_uuid: "282c7db188154044a20fcafdb7fad72a" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } }
I20260925 03:02:19.429250 9306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a9f63a6483a4a098f826dec94371591, State: Initialized, Role: FOLLOWER
I20260925 03:02:19.429678 9306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f63a6483a4a098f826dec94371591" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39067 } } peers { permanent_uuid: "31132652ea494bcca45b0f9f6e90334e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } } peers { permanent_uuid: "282c7db188154044a20fcafdb7fad72a" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } }
I20260925 03:02:19.430663 9474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f63a6483a4a098f826dec94371591" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39067 } } peers { permanent_uuid: "31132652ea494bcca45b0f9f6e90334e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } } peers { permanent_uuid: "282c7db188154044a20fcafdb7fad72a" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } } }
I20260925 03:02:19.431072 9474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 03:02:19.431833 9306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:02:19.432006 9281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:02:19.432238 9281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:19.435958 9281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f63a6483a4a098f826dec94371591" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39067 } } peers { permanent_uuid: "31132652ea494bcca45b0f9f6e90334e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } } peers { permanent_uuid: "282c7db188154044a20fcafdb7fad72a" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } }
I20260925 03:02:19.438423 9281 leader_election.cc:288] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591 [CANDIDATE]: Term 1 election: Requested vote from peers 31132652ea494bcca45b0f9f6e90334e (127.18.196.253:34079), 282c7db188154044a20fcafdb7fad72a (127.18.196.252:37443)
I20260925 03:02:19.439255 9429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6a9f63a6483a4a098f826dec94371591" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "282c7db188154044a20fcafdb7fad72a"
I20260925 03:02:19.439744 9429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 282c7db188154044a20fcafdb7fad72a [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:19.443269 9429 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 282c7db188154044a20fcafdb7fad72a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a9f63a6483a4a098f826dec94371591 in term 1.
I20260925 03:02:19.444007 9257 leader_election.cc:302] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282c7db188154044a20fcafdb7fad72a, 6a9f63a6483a4a098f826dec94371591; no voters:
I20260925 03:02:19.444499 9474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:02:19.454924 9474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591 [term 1 LEADER]: Becoming Leader. State: Replica: 6a9f63a6483a4a098f826dec94371591, State: Running, Role: LEADER
I20260925 03:02:19.455622 9474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f63a6483a4a098f826dec94371591" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39067 } } peers { permanent_uuid: "31132652ea494bcca45b0f9f6e90334e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } } peers { permanent_uuid: "282c7db188154044a20fcafdb7fad72a" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } }
I20260925 03:02:19.464111 9487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a9f63a6483a4a098f826dec94371591. Latest consensus state: current_term: 1 leader_uuid: "6a9f63a6483a4a098f826dec94371591" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f63a6483a4a098f826dec94371591" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39067 } } peers { permanent_uuid: "31132652ea494bcca45b0f9f6e90334e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } } peers { permanent_uuid: "282c7db188154044a20fcafdb7fad72a" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } } }
I20260925 03:02:19.464735 9487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591 [sys.catalog]: This master's current role is: LEADER
W20260925 03:02:19.477357 9485 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 03:02:19.477502 9485 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 03:02:19.477986 9491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:02:19.468788 9238 master_runner.cc:186] Error getting master registration for 127.18.196.253:34079: OK, instance_id { permanent_uuid: "31132652ea494bcca45b0f9f6e90334e" instance_seqno: 1790305338761466 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 03:02:19.484347 9380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } attempt: 5
I20260925 03:02:19.484711 9238 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 03:02:19.483815 9491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 03:02:19 dist-test-slave-8zvk krb5kdc[9225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305338, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.253@KRBTEST.COM for kudu/127.18.196.252@KRBTEST.COM
W20260925 03:02:19.501546 9259 tablet.cc:2369] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 03:02:19.506453 9408 tablet.cc:2369] T 00000000000000000000000000000000 P 282c7db188154044a20fcafdb7fad72a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 03:02:19.508046 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.485090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:37443 (local address 127.0.0.1:48604)
0925 03:02:19.485535 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:19.485559 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 03:02:19.485700 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:02:19.486860 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:02:19.486868 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:02:19.487565 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:02:19.487825 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:19.487841 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.489631 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.489637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:19.491122 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:19.491139 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.492760 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.492769 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:19.493034 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:19.493739 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:02:19.493762 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:02:19.496617 (+ 2855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:02:19.502102 (+ 5485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:19.502110 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:19.502116 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:19.502713 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:19.505523 (+ 2810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:19.505528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:19.505534 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:19.505777 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:19.506696 (+ 919us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:02:19.506704 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:02:19.506934 (+ 230us) client_negotiation.cc:769] Sending connection context
0925 03:02:19.507112 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 03:02:19.507510 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280}
I20260925 03:02:19.509516 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.485969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48604 (local address 127.18.196.252:37443)
0925 03:02:19.486213 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:19.486222 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:19.486245 (+ 23us) server_negotiation.cc:407] Connection header received
0925 03:02:19.486322 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:19.486330 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:19.486432 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:19.486546 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:19.488057 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.489450 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.491334 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.492082 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.492210 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:19.496832 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:19.496841 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:19.496866 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:19.496907 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:19.501905 (+ 4998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:19.505128 (+ 3223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:19.505137 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:19.505143 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:19.505356 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:19.505932 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:19.505939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:19.505945 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:19.506348 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:19.506519 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:19.508691 (+ 2172us) server_negotiation.cc:299] Negotiation successful
0925 03:02:19.508999 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 03:02:19.516942 9380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31132652ea494bcca45b0f9f6e90334e: Bootstrap starting.
I20260925 03:02:19.521952 9380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31132652ea494bcca45b0f9f6e90334e: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:19.527706 9380 log.cc:824] T 00000000000000000000000000000000 P 31132652ea494bcca45b0f9f6e90334e: Log is configured to *not* fsync() on all Append() calls
I20260925 03:02:19.546815 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.496513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47402 (local address 127.18.196.253:34079)
0925 03:02:19.496673 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:19.496679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:19.502765 (+ 6086us) server_negotiation.cc:407] Connection header received
0925 03:02:19.503065 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:19.503071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:19.503143 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:19.503212 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:19.522574 (+ 19362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.523996 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.530067 (+ 6071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.530900 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.531052 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:19.534987 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:19.534995 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:19.535002 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:19.535073 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:19.539468 (+ 4395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:19.540534 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:19.540542 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:19.540546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:19.540722 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:19.541232 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:19.541239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:19.541244 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:19.541667 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:19.541975 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:19.546082 (+ 4107us) server_negotiation.cc:299] Negotiation successful
0925 03:02:19.546356 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 03:02:19.575366 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.492427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55520 (local address 127.18.196.254:39067)
0925 03:02:19.492600 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:19.492605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:19.498161 (+ 5556us) server_negotiation.cc:407] Connection header received
0925 03:02:19.498355 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:19.498363 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:19.498449 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:19.498554 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:19.524533 (+ 25979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.525906 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.549453 (+ 23547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.550378 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.550498 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:19.563227 (+ 12729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:19.563238 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:19.563244 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:19.563307 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:19.567600 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:19.570939 (+ 3339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:19.570948 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:19.570954 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:19.571187 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:19.572143 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:19.572151 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:19.572157 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:19.572627 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:19.572946 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:19.574465 (+ 1519us) server_negotiation.cc:299] Negotiation successful
0925 03:02:19.574779 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 03:02:19.580662 9380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31132652ea494bcca45b0f9f6e90334e: No bootstrap required, opened a new log
I20260925 03:02:19.599347 9429 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 282c7db188154044a20fcafdb7fad72a [term 1 FOLLOWER]: Refusing update from remote peer 6a9f63a6483a4a098f826dec94371591: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:02:19.599551 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.507343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48608 (local address 127.18.196.252:37443)
0925 03:02:19.523596 (+ 16253us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:19.523602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:19.545723 (+ 22121us) server_negotiation.cc:407] Connection header received
0925 03:02:19.546981 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:19.546989 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:19.547100 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:19.547187 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:19.553295 (+ 6108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.554975 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.560956 (+ 5981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.561869 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.561991 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:19.587326 (+ 25335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:19.587337 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:19.587344 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:19.587397 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:19.592087 (+ 4690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:19.596747 (+ 4660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:19.596753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:19.596757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:19.596964 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:19.597449 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:19.597454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:19.597457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:19.597869 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:19.598139 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:19.598691 (+ 552us) server_negotiation.cc:299] Negotiation successful
0925 03:02:19.598965 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16125,"thread_start_us":15874,"threads_started":1}
I20260925 03:02:19.600767 9474 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282c7db188154044a20fcafdb7fad72a" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:02:19.601814 9380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31132652ea494bcca45b0f9f6e90334e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f63a6483a4a098f826dec94371591" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39067 } } peers { permanent_uuid: "31132652ea494bcca45b0f9f6e90334e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } } peers { permanent_uuid: "282c7db188154044a20fcafdb7fad72a" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } }
I20260925 03:02:19.602252 9380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31132652ea494bcca45b0f9f6e90334e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31132652ea494bcca45b0f9f6e90334e, State: Initialized, Role: FOLLOWER
I20260925 03:02:19.602761 9380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31132652ea494bcca45b0f9f6e90334e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f63a6483a4a098f826dec94371591" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39067 } } peers { permanent_uuid: "31132652ea494bcca45b0f9f6e90334e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } } peers { permanent_uuid: "282c7db188154044a20fcafdb7fad72a" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } }
I20260925 03:02:19.604095 9500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31132652ea494bcca45b0f9f6e90334e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f63a6483a4a098f826dec94371591" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39067 } } peers { permanent_uuid: "31132652ea494bcca45b0f9f6e90334e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } } peers { permanent_uuid: "282c7db188154044a20fcafdb7fad72a" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } } }
I20260925 03:02:19.604889 9500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31132652ea494bcca45b0f9f6e90334e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 03:02:19.605204 9380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31132652ea494bcca45b0f9f6e90334e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:02:19.605551 9354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31132652ea494bcca45b0f9f6e90334e [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:19.607170 9355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6a9f63a6483a4a098f826dec94371591" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "31132652ea494bcca45b0f9f6e90334e"
I20260925 03:02:19.611155 9354 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 31132652ea494bcca45b0f9f6e90334e [term 1 FOLLOWER]: Refusing update from remote peer 6a9f63a6483a4a098f826dec94371591: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:02:19.615864 9355 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 31132652ea494bcca45b0f9f6e90334e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a9f63a6483a4a098f826dec94371591 in term 1.
I20260925 03:02:19.618731 9474 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31132652ea494bcca45b0f9f6e90334e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:02:19.633267 9460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282c7db188154044a20fcafdb7fad72a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a9f63a6483a4a098f826dec94371591. Latest consensus state: current_term: 1 leader_uuid: "6a9f63a6483a4a098f826dec94371591" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f63a6483a4a098f826dec94371591" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39067 } } peers { permanent_uuid: "31132652ea494bcca45b0f9f6e90334e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } } peers { permanent_uuid: "282c7db188154044a20fcafdb7fad72a" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } } }
I20260925 03:02:19.633674 9460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282c7db188154044a20fcafdb7fad72a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 03:02:19.637202 9496 mvcc.cc:205] Tried to move back new op lower bound from 7333090670984519680 to 7333090670416547840. Current Snapshot: MvccSnapshot[applied={T|T < 7333090670984519680}]
I20260925 03:02:19.653614 9501 mvcc.cc:205] Tried to move back new op lower bound from 7333090670984519680 to 7333090670416547840. Current Snapshot: MvccSnapshot[applied={T|T < 7333090670984519680}]
I20260925 03:02:19.654201 9500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31132652ea494bcca45b0f9f6e90334e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a9f63a6483a4a098f826dec94371591. Latest consensus state: current_term: 1 leader_uuid: "6a9f63a6483a4a098f826dec94371591" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f63a6483a4a098f826dec94371591" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39067 } } peers { permanent_uuid: "31132652ea494bcca45b0f9f6e90334e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } } peers { permanent_uuid: "282c7db188154044a20fcafdb7fad72a" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } } }
I20260925 03:02:19.654583 9500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31132652ea494bcca45b0f9f6e90334e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 03:02:19.665330 9504 mvcc.cc:205] Tried to move back new op lower bound from 7333090670984519680 to 7333090670416547840. Current Snapshot: MvccSnapshot[applied={T|T < 7333090670984519680}]
I20260925 03:02:19.670579 9487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6a9f63a6483a4a098f826dec94371591" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f63a6483a4a098f826dec94371591" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39067 } } peers { permanent_uuid: "31132652ea494bcca45b0f9f6e90334e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } } peers { permanent_uuid: "282c7db188154044a20fcafdb7fad72a" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } } }
I20260925 03:02:19.670993 9487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591 [sys.catalog]: This master's current role is: LEADER
I20260925 03:02:19.671478 9487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6a9f63a6483a4a098f826dec94371591" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f63a6483a4a098f826dec94371591" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39067 } } peers { permanent_uuid: "31132652ea494bcca45b0f9f6e90334e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } } peers { permanent_uuid: "282c7db188154044a20fcafdb7fad72a" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } } }
I20260925 03:02:19.671720 9487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591 [sys.catalog]: This master's current role is: LEADER
I20260925 03:02:19.674722 9491 catalog_manager.cc:1347] Generated new cluster ID: a09b416d6989417c86dbfaa01abd5d3f
I20260925 03:02:19.674814 9491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:02:19.675177 9500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31132652ea494bcca45b0f9f6e90334e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6a9f63a6483a4a098f826dec94371591" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f63a6483a4a098f826dec94371591" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39067 } } peers { permanent_uuid: "31132652ea494bcca45b0f9f6e90334e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } } peers { permanent_uuid: "282c7db188154044a20fcafdb7fad72a" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } } }
I20260925 03:02:19.675407 9500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31132652ea494bcca45b0f9f6e90334e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 03:02:19.676324 9460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282c7db188154044a20fcafdb7fad72a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6a9f63a6483a4a098f826dec94371591" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9f63a6483a4a098f826dec94371591" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 39067 } } peers { permanent_uuid: "31132652ea494bcca45b0f9f6e90334e" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 34079 } } peers { permanent_uuid: "282c7db188154044a20fcafdb7fad72a" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 37443 } } }
I20260925 03:02:19.676573 9460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282c7db188154044a20fcafdb7fad72a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 03:02:19.702750 9491 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:02:19.704478 9491 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:02:19.725565 9491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a9f63a6483a4a098f826dec94371591: Generated new TSK 0
I20260925 03:02:19.726759 9491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:02:19.761447 9518 catalog_manager.cc:1259] Loaded cluster ID: a09b416d6989417c86dbfaa01abd5d3f
I20260925 03:02:19.761633 9518 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 31132652ea494bcca45b0f9f6e90334e: loading cluster ID for follower catalog manager: success
I20260925 03:02:19.765388 9518 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 31132652ea494bcca45b0f9f6e90334e: acquiring CA information for follower catalog manager: success
I20260925 03:02:19.767968 9518 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 31132652ea494bcca45b0f9f6e90334e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 03:02:19.883270 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 03:02:19.873502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47414 (local address 127.18.196.253:34079)
0925 03:02:19.873690 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:19.873696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:19.879533 (+ 5837us) server_negotiation.cc:407] Connection header received
0925 03:02:19.881144 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:19.881150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:19.881265 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:19.881350 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:19.882992 (+ 1642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47414: BlockingRecv error: recv got EOF from 127.0.0.1:47414 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 03:02:19.885066 9310 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 03:02:19.865146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55528 (local address 127.18.196.254:39067)
0925 03:02:19.865347 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:19.865353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:19.879073 (+ 13720us) server_negotiation.cc:407] Connection header received
0925 03:02:19.881593 (+ 2520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:19.881599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:19.881705 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:19.881781 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:19.884814 (+ 3033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55528: BlockingRecv error: recv got EOF from 127.0.0.1:55528 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 03:02:19.887583 9497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 03:02:19.880052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.18.196.252:37443)
0925 03:02:19.880246 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:19.880253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:19.885985 (+ 5732us) server_negotiation.cc:407] Connection header received
0925 03:02:19.886323 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:19.886329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:19.886406 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:19.886476 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:19.887346 (+ 870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48614: BlockingRecv error: recv got EOF from 127.0.0.1:48614 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 03:02:19.946712 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:19.928820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47424 (local address 127.18.196.253:34079)
0925 03:02:19.929058 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:19.929068 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:19.939012 (+ 9944us) server_negotiation.cc:407] Connection header received
0925 03:02:19.939367 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:19.939374 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:19.939611 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:19.939743 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:19.940497 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.941560 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.943674 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:19.944339 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:19.944444 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:19.945176 (+ 732us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:19.945846 (+ 670us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:19.945962 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:19.946263 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 03:02:19.946360 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 03:02:19.992550 9497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 03:02:19.983454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48616 (local address 127.18.196.252:37443)
0925 03:02:19.983640 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:19.983646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:19.990427 (+ 6781us) server_negotiation.cc:407] Connection header received
0925 03:02:19.990976 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:19.990983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:19.991144 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:19.991238 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:19.992284 (+ 1046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48616: BlockingRecv error: recv got EOF from 127.0.0.1:48616 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 03:02:20.040560 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:20.026415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47430 (local address 127.18.196.253:34079)
0925 03:02:20.026865 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:20.026872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:20.033680 (+ 6808us) server_negotiation.cc:407] Connection header received
0925 03:02:20.034145 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:20.034152 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:20.034263 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:20.034327 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:20.035050 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:20.036266 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:20.038286 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:20.038964 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:20.039065 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:20.039724 (+ 659us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:20.039885 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:20.039943 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:20.040176 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 03:02:20.040258 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293}
W20260925 03:02:20.054738 9497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 03:02:20.052492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48630 (local address 127.18.196.252:37443)
0925 03:02:20.052717 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:20.052726 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:20.053267 (+ 541us) server_negotiation.cc:407] Connection header received
0925 03:02:20.053542 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:20.053548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:20.053631 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:20.053740 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:20.054508 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48630: BlockingRecv error: recv got EOF from 127.0.0.1:48630 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 03:02:20.065310 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:20.059756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47442 (local address 127.18.196.253:34079)
0925 03:02:20.059969 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:20.059978 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:20.060612 (+ 634us) server_negotiation.cc:407] Connection header received
0925 03:02:20.060821 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:20.060827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:20.060903 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:20.061009 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:20.061429 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:20.062194 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:20.063572 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:20.064021 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:20.064162 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:20.064535 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:20.064657 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:20.064753 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:20.064895 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 03:02:20.064972 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 03:02:20.073650 9497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 03:02:20.071503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48638 (local address 127.18.196.252:37443)
0925 03:02:20.071787 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:20.071796 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:20.072292 (+ 496us) server_negotiation.cc:407] Connection header received
0925 03:02:20.072517 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:20.072521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:20.072597 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:20.072710 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:20.073471 (+ 761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48638: BlockingRecv error: recv got EOF from 127.0.0.1:48638 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 03:02:20.087222 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:20.081432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47458 (local address 127.18.196.253:34079)
0925 03:02:20.081664 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:20.081674 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:20.082228 (+ 554us) server_negotiation.cc:407] Connection header received
0925 03:02:20.082445 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:20.082451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:20.082529 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:20.082645 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:20.083110 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:20.083931 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:20.085376 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:20.085851 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:20.085999 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:20.086419 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:20.086531 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:20.086648 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:20.086796 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 03:02:20.086908 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 03:02:20.095718 9497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 03:02:20.092946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.18.196.252:37443)
0925 03:02:20.093225 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:20.093233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:20.094315 (+ 1082us) server_negotiation.cc:407] Connection header received
0925 03:02:20.094527 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:20.094533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:20.094616 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:20.094724 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:20.095481 (+ 757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48644: BlockingRecv error: recv got EOF from 127.0.0.1:48644 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
I20260925 03:02:20.112291 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:20.106685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47472 (local address 127.18.196.253:34079)
0925 03:02:20.106980 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:20.106989 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:20.107483 (+ 494us) server_negotiation.cc:407] Connection header received
0925 03:02:20.107697 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:20.107705 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:20.107783 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:20.107880 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:20.108295 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:20.109050 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:20.110365 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:20.110893 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:20.111034 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:20.111450 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:20.111580 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:20.111684 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:20.111850 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 03:02:20.111946 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 03:02:20.121519 9497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 03:02:20.119177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48656 (local address 127.18.196.252:37443)
0925 03:02:20.119365 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:20.119373 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:20.119893 (+ 520us) server_negotiation.cc:407] Connection header received
0925 03:02:20.120181 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:20.120186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:20.120290 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:20.120404 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:20.121281 (+ 877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48656: BlockingRecv error: recv got EOF from 127.0.0.1:48656 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 03:02:20.150383 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:20.144687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47484 (local address 127.18.196.253:34079)
0925 03:02:20.144927 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:20.144937 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:20.145481 (+ 544us) server_negotiation.cc:407] Connection header received
0925 03:02:20.145709 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:20.145713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:20.145781 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:20.145884 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:20.146303 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:20.147229 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:20.148610 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:20.149040 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:20.149175 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:20.149585 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:20.149710 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:20.149822 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:20.149989 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 03:02:20.150079 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 03:02:20.158237 9497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 03:02:20.156351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48668 (local address 127.18.196.252:37443)
0925 03:02:20.156552 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:20.156560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:20.156886 (+ 326us) server_negotiation.cc:407] Connection header received
0925 03:02:20.157025 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:20.157032 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:20.157136 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:20.157257 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:20.158011 (+ 754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48668: BlockingRecv error: recv got EOF from 127.0.0.1:48668 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 03:02:20.200630 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:20.194251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47500 (local address 127.18.196.253:34079)
0925 03:02:20.194467 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:20.194474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:20.194867 (+ 393us) server_negotiation.cc:407] Connection header received
0925 03:02:20.195117 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:20.195124 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:20.195239 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:20.195374 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:20.195808 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:20.197005 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:20.198413 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:20.199197 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:20.199358 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:20.199714 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:20.199850 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:20.199970 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:20.200173 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 03:02:20.200272 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260925 03:02:20.209198 9497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 03:02:20.207259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48674 (local address 127.18.196.252:37443)
0925 03:02:20.207456 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:20.207463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:20.207847 (+ 384us) server_negotiation.cc:407] Connection header received
0925 03:02:20.207980 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:20.207986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:20.208091 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:20.208209 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:20.208934 (+ 725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48674: BlockingRecv error: recv got EOF from 127.0.0.1:48674 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 03:02:20.283226 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:20.276842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47502 (local address 127.18.196.253:34079)
0925 03:02:20.276999 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:20.277005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:20.277502 (+ 497us) server_negotiation.cc:407] Connection header received
0925 03:02:20.277675 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:20.277681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:20.277765 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:20.277854 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:20.278314 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:20.279437 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:20.281291 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:20.281885 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:20.282007 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:20.282481 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:20.282602 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:20.282684 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:20.282861 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 03:02:20.282940 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 03:02:20.290723 9497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 03:02:20.288782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48680 (local address 127.18.196.252:37443)
0925 03:02:20.289044 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:20.289051 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:20.289369 (+ 318us) server_negotiation.cc:407] Connection header received
0925 03:02:20.289525 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:20.289529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:20.289598 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:20.289683 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:20.290516 (+ 833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48680: BlockingRecv error: recv got EOF from 127.0.0.1:48680 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 03:02:20.362598 9471 catalog_manager.cc:1259] Loaded cluster ID: a09b416d6989417c86dbfaa01abd5d3f
I20260925 03:02:20.362726 9471 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 282c7db188154044a20fcafdb7fad72a: loading cluster ID for follower catalog manager: success
I20260925 03:02:20.365166 9471 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 282c7db188154044a20fcafdb7fad72a: acquiring CA information for follower catalog manager: success
I20260925 03:02:20.366926 9471 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 282c7db188154044a20fcafdb7fad72a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 03:02:20.426954 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:20.421952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47510 (local address 127.18.196.253:34079)
0925 03:02:20.422126 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:20.422130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:20.422522 (+ 392us) server_negotiation.cc:407] Connection header received
0925 03:02:20.422687 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:20.422691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:20.422759 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:20.422941 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:20.423274 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:20.423964 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:20.425372 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:20.425792 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:20.425884 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:20.426235 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:20.426339 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:20.426404 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:20.426579 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 03:02:20.426650 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 03:02:20.438292 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:20.432745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48686 (local address 127.18.196.252:37443)
0925 03:02:20.432993 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:20.432999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:20.433200 (+ 201us) server_negotiation.cc:407] Connection header received
0925 03:02:20.433399 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:20.433403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:20.433552 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:20.433644 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:20.433961 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:20.434671 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:20.436132 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:20.436524 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:20.436617 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:20.437118 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:20.437681 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:20.437796 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:20.437860 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 03:02:20.437995 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 03:02:20.457746 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:20.446047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55534 (local address 127.18.196.254:39067)
0925 03:02:20.451238 (+ 5191us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:20.451243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:20.451258 (+ 15us) server_negotiation.cc:407] Connection header received
0925 03:02:20.451357 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:20.451360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:20.451506 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:20.451657 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:20.452320 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:20.453578 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:20.455599 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:20.455999 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:20.456133 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:20.456680 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:20.457111 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:20.457229 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:20.457334 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 03:02:20.457419 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5097,"thread_start_us":4987,"threads_started":1}
I20260925 03:02:20.458410 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:20.445845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47526 (local address 127.18.196.253:34079)
0925 03:02:20.446065 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:20.446072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:20.452378 (+ 6306us) server_negotiation.cc:407] Connection header received
0925 03:02:20.452500 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:20.452506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:20.452585 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:20.452681 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:20.453239 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:20.454306 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:20.456465 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:20.456931 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:20.457072 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:20.457559 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:20.457682 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:20.457761 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:20.458120 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 03:02:20.458215 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 03:02:20.460306 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:20.452465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48692 (local address 127.18.196.252:37443)
0925 03:02:20.452639 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:20.452644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:20.453738 (+ 1094us) server_negotiation.cc:407] Connection header received
0925 03:02:20.453920 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:20.453926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:20.454033 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:20.454117 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:20.454986 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:20.456070 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:20.458052 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:20.458614 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:20.458710 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:20.459402 (+ 692us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:20.459530 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:20.459597 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:20.459863 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 03:02:20.459989 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 03:02:20.463318 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 9238
I20260925 03:02:20.464378 9301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:02:20.572408 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 9238
I20260925 03:02:20.593119 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 9313
I20260925 03:02:20.594218 9375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:02:20.695921 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 9313
I20260925 03:02:20.715444 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 9387
I20260925 03:02:20.716480 9449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:02:20.819283 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 9387
2026-09-25T03:02:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4288 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 03:02:20 dist-test-slave-8zvk krb5kdc[9633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 03:02:20 dist-test-slave-8zvk krb5kdc[9633](info): set up 2 sockets
Sep 25 03:02:20 dist-test-slave-8zvk krb5kdc[9633](info): commencing operation
krb5kdc: starting...
W20260925 03:02:21.918758 19219 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/krb5kdc/test-user.keytab.
Sep 25 03:02:22 dist-test-slave-8zvk krb5kdc[9633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T03:02:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T03:02:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
Entry for principal kudu/127.18.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab.
I20260925 03:02:22.160004 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:43313
--webserver_interface=127.18.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:40263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:43313,127.18.196.253:41465,127.18.196.252:41927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 03:02:22.482523 9646 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 03:02:22.482872 9646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:02:22.482997 9646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:02:22.483127 9646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:02:22.490124 9646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:02:22.490200 9646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:02:22.490312 9646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:02:22.490404 9646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:02:22.490520 9646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:02:22.500257 9646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:40263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:43313,127.18.196.253:41465,127.18.196.252:41927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.254.keytab
--principal=kudu/127.18.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.254:43313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:02:22.501550 9646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:02:22.502928 9646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:02:22.514493 9651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:22.514531 9654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:22.514572 9652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:02:22.515257 9646 server_base.cc:1034] running on GCE node
I20260925 03:02:22.515995 9646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:02:22.517450 9646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:02:22.518803 9646 hybrid_clock.cc:648] HybridClock initialized: now 1790305342518748 us; error 50 us; skew 500 ppm
I20260925 03:02:22.519220 9646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:02:22 dist-test-slave-8zvk krb5kdc[9633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305342, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:02:22.526292 9646 init.cc:377] Logged in from keytab as kudu/127.18.196.254@KRBTEST.COM (short username kudu)
I20260925 03:02:22.527678 9646 webserver.cc:460] Webserver started at http://127.18.196.254:45653/ using document root <none> and password file <none>
I20260925 03:02:22.528347 9646 fs_manager.cc:359] Metadata directory not provided
I20260925 03:02:22.528475 9646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:02:22.528786 9646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:02:22.531736 9646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-0/data/instance:
uuid: "ca53c07fba764f5395d6d814c7e72ae5"
format_stamp: "Formatted at 2026-09-25 03:02:22 on dist-test-slave-8zvk"
server_key: "bb779f3e2a61bdfd796a9bf9cf726cb2"
server_key_iv: "73aade4a951bbe6666190252630270fb"
server_key_version: "encryptionkey@0"
I20260925 03:02:22.532511 9646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-0/wal/instance:
uuid: "ca53c07fba764f5395d6d814c7e72ae5"
format_stamp: "Formatted at 2026-09-25 03:02:22 on dist-test-slave-8zvk"
server_key: "bb779f3e2a61bdfd796a9bf9cf726cb2"
server_key_iv: "73aade4a951bbe6666190252630270fb"
server_key_version: "encryptionkey@0"
I20260925 03:02:22.537312 9646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 03:02:22.540504 9661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:22.541749 9646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 03:02:22.541952 9646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-0/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-0/wal
uuid: "ca53c07fba764f5395d6d814c7e72ae5"
format_stamp: "Formatted at 2026-09-25 03:02:22 on dist-test-slave-8zvk"
server_key: "bb779f3e2a61bdfd796a9bf9cf726cb2"
server_key_iv: "73aade4a951bbe6666190252630270fb"
server_key_version: "encryptionkey@0"
I20260925 03:02:22.542167 9646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:02:22.571350 9646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:02:22.574995 9646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:02:22.575266 9646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:02:22.593364 9646 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.254:43313
I20260925 03:02:22.593389 9712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.254:43313 every 8 connection(s)
I20260925 03:02:22.594646 9646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-0/data/info.pb
I20260925 03:02:22.598230 9713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:22.603561 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 9646
I20260925 03:02:22.603739 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-0/wal/instance
I20260925 03:02:22.604228 19219 external_mini_cluster.cc:1428] Setting key 915db514004b97d75340b1d3e5584698
I20260925 03:02:22.605129 9713 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43313 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 03:02:22 dist-test-slave-8zvk krb5kdc[9633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305342, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:02:22.629011 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:22.608963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56174 (local address 127.18.196.254:43313)
0925 03:02:22.610559 (+ 1596us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:22.610573 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:22.610619 (+ 46us) server_negotiation.cc:407] Connection header received
0925 03:02:22.612182 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:22.612187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:22.612402 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:22.612610 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:22.613847 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:22.615103 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:22.617332 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:22.617869 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:22.618017 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:22.621975 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:22.621983 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:22.621994 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:22.622021 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:22.624766 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:22.625466 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:22.625474 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:22.625478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:22.625619 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:22.625999 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:22.626006 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:22.626009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:22.626473 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:22.626682 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:22.627236 (+ 554us) server_negotiation.cc:299] Negotiation successful
0925 03:02:22.627583 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1364,"thread_start_us":1164,"threads_started":1}
I20260925 03:02:22.629051 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:22.608962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:43313 (local address 127.0.0.1:56174)
0925 03:02:22.609766 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:22.609816 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 03:02:22.611828 (+ 2012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:02:22.612614 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:02:22.612622 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:02:22.613303 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:02:22.613628 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:22.613646 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:22.615261 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:22.615269 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:22.617180 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:22.617198 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:22.618151 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:22.618160 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:22.618376 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:22.619134 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:02:22.619167 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:02:22.621712 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:02:22.624913 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:22.624927 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:22.624941 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:22.625317 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:22.625729 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:22.625734 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:22.625738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:22.625890 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:22.626603 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:02:22.626610 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:02:22.626820 (+ 210us) client_negotiation.cc:769] Sending connection context
0925 03:02:22.627121 (+ 301us) client_negotiation.cc:240] Negotiation successful
0925 03:02:22.627696 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":144,"thread_start_us":347,"threads_started":1}
WARNING: no policy specified for kudu/127.18.196.253@KRBTEST.COM; defaulting to no policy
I20260925 03:02:22.632157 9713 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } has no permanent_uuid. Determining permanent_uuid...
I20260925 03:02:22.633268 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:22.632835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:41465 (local address 127.0.0.1:53536)
0925 03:02:22.632989 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:22.633155 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 03:02:22.633566 9662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.196.253:41465: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.253@KRBTEST.COM" created.
I20260925 03:02:22.635262 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:22.634763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:41465 (local address 127.0.0.1:53546)
0925 03:02:22.634984 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:22.635075 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 03:02:22.635538 9713 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.253:41465: Network error: Client connection negotiation failed: client connection to 127.18.196.253:41465: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.253.keytab.
Entry for principal kudu/127.18.196.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.253.keytab.
I20260925 03:02:22.664155 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.253:41465
--webserver_interface=127.18.196.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.253.keytab
--principal=kudu/127.18.196.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:40263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:43313,127.18.196.253:41465,127.18.196.252:41927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 03:02:22.684993 9713 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } attempt: 1
I20260925 03:02:22.686239 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:22.685792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:41465 (local address 127.0.0.1:53552)
0925 03:02:22.685997 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:22.686113 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 03:02:22.687397 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:22.687025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:41465 (local address 127.0.0.1:53554)
0925 03:02:22.687208 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:22.687270 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260925 03:02:22.687577 9713 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.253:41465: Network error: Client connection negotiation failed: client connection to 127.18.196.253:41465: connect: Connection refused (error 111)
I20260925 03:02:22.755829 9713 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } attempt: 2
I20260925 03:02:22.757009 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:22.756524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:41465 (local address 127.0.0.1:53570)
0925 03:02:22.756781 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:22.756870 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 03:02:22.758049 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:22.757689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:41465 (local address 127.0.0.1:53582)
0925 03:02:22.757853 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:22.757918 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 03:02:22.758239 9713 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.253:41465: Network error: Client connection negotiation failed: client connection to 127.18.196.253:41465: connect: Connection refused (error 111)
I20260925 03:02:22.849483 9713 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } attempt: 3
I20260925 03:02:22.850600 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:22.850158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:41465 (local address 127.0.0.1:53598)
0925 03:02:22.850379 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:22.850468 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 03:02:22.851756 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:22.851379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:41465 (local address 127.0.0.1:53608)
0925 03:02:22.851531 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:22.851603 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 03:02:22.851974 9713 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.253:41465: Network error: Client connection negotiation failed: client connection to 127.18.196.253:41465: connect: Connection refused (error 111)
W20260925 03:02:22.987684 9720 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 03:02:22.988055 9720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:02:22.988183 9720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:02:22.988320 9720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 03:02:22.995222 9713 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } attempt: 4
W20260925 03:02:22.995373 9720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:02:22.995507 9720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:02:22.995638 9720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:02:22.995718 9720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:02:22.995818 9720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:02:22.996510 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:22.995954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:41465 (local address 127.0.0.1:53620)
0925 03:02:22.996285 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:22.996378 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260925 03:02:22.997625 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:22.997262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:41465 (local address 127.0.0.1:53626)
0925 03:02:22.997440 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:22.997513 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.253:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 03:02:22.997805 9713 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.253:41465: Network error: Client connection negotiation failed: client connection to 127.18.196.253:41465: connect: Connection refused (error 111)
I20260925 03:02:23.006325 9720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:40263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:43313,127.18.196.253:41465,127.18.196.252:41927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.253.keytab
--principal=kudu/127.18.196.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.253:41465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.196.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:02:23.007565 9720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:02:23.008733 9720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:02:23.020032 9726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:23.020068 9728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:23.019991 9725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:02:23.021166 9720 server_base.cc:1034] running on GCE node
I20260925 03:02:23.021853 9720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:02:23.023216 9720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:02:23.024467 9720 hybrid_clock.cc:648] HybridClock initialized: now 1790305343024387 us; error 32 us; skew 500 ppm
I20260925 03:02:23.024780 9720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:02:23 dist-test-slave-8zvk krb5kdc[9633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305343, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:02:23.027777 9720 init.cc:377] Logged in from keytab as kudu/127.18.196.253@KRBTEST.COM (short username kudu)
I20260925 03:02:23.029286 9720 webserver.cc:460] Webserver started at http://127.18.196.253:45541/ using document root <none> and password file <none>
I20260925 03:02:23.029949 9720 fs_manager.cc:359] Metadata directory not provided
I20260925 03:02:23.030081 9720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:02:23.030390 9720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:02:23.033216 9720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-1/data/instance:
uuid: "9a8ee6fdb2e946f096ed94d56a4fea62"
format_stamp: "Formatted at 2026-09-25 03:02:23 on dist-test-slave-8zvk"
server_key: "ff79c409ff938dfd63cb4db2df6f37cd"
server_key_iv: "0b97c40a1d60a705078649b64d01bbe0"
server_key_version: "encryptionkey@0"
I20260925 03:02:23.033972 9720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-1/wal/instance:
uuid: "9a8ee6fdb2e946f096ed94d56a4fea62"
format_stamp: "Formatted at 2026-09-25 03:02:23 on dist-test-slave-8zvk"
server_key: "ff79c409ff938dfd63cb4db2df6f37cd"
server_key_iv: "0b97c40a1d60a705078649b64d01bbe0"
server_key_version: "encryptionkey@0"
I20260925 03:02:23.038630 9720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260925 03:02:23.041847 9735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:23.043131 9720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 03:02:23.043294 9720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-1/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-1/wal
uuid: "9a8ee6fdb2e946f096ed94d56a4fea62"
format_stamp: "Formatted at 2026-09-25 03:02:23 on dist-test-slave-8zvk"
server_key: "ff79c409ff938dfd63cb4db2df6f37cd"
server_key_iv: "0b97c40a1d60a705078649b64d01bbe0"
server_key_version: "encryptionkey@0"
I20260925 03:02:23.043483 9720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:02:23.061894 9720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:02:23.065835 9720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:02:23.066103 9720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:02:23.083335 9720 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.253:41465
I20260925 03:02:23.083366 9786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.253:41465 every 8 connection(s)
I20260925 03:02:23.084560 9720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-1/data/info.pb
I20260925 03:02:23.088060 9787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:23.088781 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 9720
I20260925 03:02:23.088975 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-1/wal/instance
I20260925 03:02:23.089524 19219 external_mini_cluster.cc:1428] Setting key d553ee23d5b9a7d749e16798f5451de7
I20260925 03:02:23.098588 9787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43313 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 03:02:23 dist-test-slave-8zvk krb5kdc[9633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305343, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.253@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:02:23.119683 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.102202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56190 (local address 127.18.196.254:43313)
0925 03:02:23.102449 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:23.102458 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:23.103564 (+ 1106us) server_negotiation.cc:407] Connection header received
0925 03:02:23.105427 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:23.105432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:23.105527 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:23.105638 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:23.107126 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.107806 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.109261 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.109704 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.109838 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:23.113743 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:23.113748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:23.113751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:23.113781 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:23.116580 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.117392 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.117397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.117399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.117528 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.117944 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.117947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.117949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.118221 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:23.118389 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:23.119126 (+ 737us) server_negotiation.cc:299] Negotiation successful
0925 03:02:23.119355 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
WARNING: no policy specified for kudu/127.18.196.252@KRBTEST.COM; defaulting to no policy
I20260925 03:02:23.120599 9790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.102574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:43313 (local address 127.0.0.1:56190)
0925 03:02:23.103382 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.103445 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 03:02:23.105165 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:02:23.105862 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:02:23.105870 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:02:23.106359 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:02:23.106963 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:23.106981 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.107962 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.107970 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:23.109106 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:23.109118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.109919 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.109927 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:23.110158 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:23.110873 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:02:23.110909 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:02:23.113526 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:02:23.116724 (+ 3198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:23.116730 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:23.116774 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:23.117240 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:23.117642 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:23.117647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:23.117651 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:23.117814 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:23.118358 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:02:23.118367 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:02:23.118709 (+ 342us) client_negotiation.cc:769] Sending connection context
0925 03:02:23.119022 (+ 313us) client_negotiation.cc:240] Negotiation successful
0925 03:02:23.119507 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":367,"threads_started":1}
I20260925 03:02:23.122239 9787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.196.252@KRBTEST.COM" created.
Sep 25 03:02:23 dist-test-slave-8zvk krb5kdc[9633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305343, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.253@KRBTEST.COM for kudu/127.18.196.253@KRBTEST.COM
I20260925 03:02:23.136791 9791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.123188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53640 (local address 127.18.196.253:41465)
0925 03:02:23.124019 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:23.124027 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:23.124079 (+ 52us) server_negotiation.cc:407] Connection header received
0925 03:02:23.124120 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:23.124125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:23.124280 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:23.124437 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:23.125135 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.126106 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.127257 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.127778 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.127902 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:23.131125 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:23.131143 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:23.131161 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:23.131203 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:23.133930 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.134712 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.134718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.134720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.134823 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.135281 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.135284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.135286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.135596 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:23.135743 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:23.136199 (+ 456us) server_negotiation.cc:299] Negotiation successful
0925 03:02:23.136498 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":649,"thread_start_us":493,"threads_started":1}
I20260925 03:02:23.136870 9790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.123098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:41465 (local address 127.0.0.1:53640)
0925 03:02:23.123273 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.123294 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 03:02:23.123427 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:02:23.124418 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:02:23.124423 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:02:23.124841 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:02:23.124972 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:23.124980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.126233 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.126240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:23.127129 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:23.127140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.127951 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.127956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:23.128113 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:23.128670 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:02:23.128686 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:02:23.130814 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:02:23.134066 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:23.134073 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:23.134080 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:23.134590 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:23.134973 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:23.134985 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:23.134991 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:23.135171 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:23.135727 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:02:23.135735 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:02:23.135906 (+ 171us) client_negotiation.cc:769] Sending connection context
0925 03:02:23.136060 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 03:02:23.136382 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 03:02:23.139148 9787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } has no permanent_uuid. Determining permanent_uuid...
I20260925 03:02:23.140434 9790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.139884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:41927 (local address 127.0.0.1:48582)
0925 03:02:23.140105 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.140240 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 03:02:23.140707 9737 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
I20260925 03:02:23.142341 9790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.141903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:41927 (local address 127.0.0.1:48586)
0925 03:02:23.142095 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.142199 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 03:02:23.142555 9787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:41927: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.196.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.252.keytab.
Entry for principal kudu/127.18.196.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.252.keytab.
I20260925 03:02:23.155772 19219 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
/tmp/dist-test-taskM3GXH0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.252:41927
--webserver_interface=127.18.196.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.252.keytab
--principal=kudu/127.18.196.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.196.212:40263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.196.254:43313,127.18.196.253:41465,127.18.196.252:41927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 03:02:23.192129 9787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } attempt: 1
I20260925 03:02:23.193404 9790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.192853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:41927 (local address 127.0.0.1:48594)
0925 03:02:23.193089 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.193218 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 03:02:23.194454 9790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.194066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:41927 (local address 127.0.0.1:48602)
0925 03:02:23.194233 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.194327 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 03:02:23.194674 9787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:41927: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
I20260925 03:02:23.262926 9787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } attempt: 2
I20260925 03:02:23.264138 9790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.263606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:41927 (local address 127.0.0.1:48608)
0925 03:02:23.263827 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.263946 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 03:02:23.265233 9790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.264788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:41927 (local address 127.0.0.1:48620)
0925 03:02:23.264973 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.265071 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 03:02:23.265486 9787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:41927: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
I20260925 03:02:23.297050 9713 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } attempt: 5
Sep 25 03:02:23 dist-test-slave-8zvk krb5kdc[9633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305342, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for kudu/127.18.196.253@KRBTEST.COM
I20260925 03:02:23.312037 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.297856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:41465 (local address 127.0.0.1:53656)
0925 03:02:23.298083 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.298117 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 03:02:23.298250 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:02:23.298642 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:02:23.298646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:02:23.299068 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:02:23.299265 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:23.299274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.300742 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.300747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:23.301586 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:23.301604 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.302783 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.302787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:23.303016 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:23.303639 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:02:23.303656 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:02:23.305783 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:02:23.309582 (+ 3799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:23.309590 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:23.309596 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:23.310073 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:23.310559 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:23.310563 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:23.310565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:23.310699 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:23.311356 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:02:23.311360 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:02:23.311482 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 03:02:23.311604 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 03:02:23.311813 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 03:02:23.312256 9791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.298098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53656 (local address 127.18.196.253:41465)
0925 03:02:23.298304 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:23.298311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:23.298332 (+ 21us) server_negotiation.cc:407] Connection header received
0925 03:02:23.298415 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:23.298420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:23.298497 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:23.298604 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:23.299427 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.300591 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.301755 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.302617 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.302779 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:23.305989 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:23.305996 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:23.306001 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:23.306039 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:23.309391 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.310207 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.310215 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.310220 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.310403 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.310816 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.310823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.310826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.311219 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:23.311358 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:23.311741 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 03:02:23.311963 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 03:02:23.313278 9713 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } has no permanent_uuid. Determining permanent_uuid...
I20260925 03:02:23.314296 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.313965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:41927 (local address 127.0.0.1:48624)
0925 03:02:23.314114 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.314184 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 03:02:23.315269 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.314951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:41927 (local address 127.0.0.1:48634)
0925 03:02:23.315110 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.315170 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 03:02:23.315496 9713 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:41927: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
I20260925 03:02:23.356745 9787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } attempt: 3
I20260925 03:02:23.357936 9790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.357413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:41927 (local address 127.0.0.1:48636)
0925 03:02:23.357632 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.357751 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 03:02:23.359125 9790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.358609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:41927 (local address 127.0.0.1:48648)
0925 03:02:23.358868 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.358971 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260925 03:02:23.359400 9787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:41927: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
I20260925 03:02:23.366708 9713 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } attempt: 1
I20260925 03:02:23.367818 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.367264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:41927 (local address 127.0.0.1:48660)
0925 03:02:23.367488 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.367606 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 03:02:23.368973 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.368518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:41927 (local address 127.0.0.1:48662)
0925 03:02:23.368698 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.368798 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 03:02:23.369268 9713 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:41927: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
I20260925 03:02:23.437491 9713 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } attempt: 2
I20260925 03:02:23.438701 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.438137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:41927 (local address 127.0.0.1:48676)
0925 03:02:23.438388 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.438501 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 03:02:23.439905 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.439453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:41927 (local address 127.0.0.1:48678)
0925 03:02:23.439626 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.439732 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 03:02:23.440197 9713 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:41927: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
W20260925 03:02:23.488181 9794 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 03:02:23.488526 9794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 03:02:23.488651 9794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 03:02:23.488766 9794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 03:02:23.496534 9794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 03:02:23.496634 9794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 03:02:23.496737 9794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 03:02:23.496836 9794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 03:02:23.496948 9794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 03:02:23.502633 9787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } attempt: 4
I20260925 03:02:23.503836 9790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.503277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:41927 (local address 127.0.0.1:48686)
0925 03:02:23.503541 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.503649 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 03:02:23.504896 9790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.504537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:41927 (local address 127.0.0.1:48690)
0925 03:02:23.504688 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.504772 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 03:02:23.505106 9787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:41927: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
I20260925 03:02:23.507674 9794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.196.212:40263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.196.254:43313,127.18.196.253:41465,127.18.196.252:41927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/krb5kdc/kudu_127.18.196.252.keytab
--principal=kudu/127.18.196.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.196.252:41927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.196.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 03:02:23.508930 9794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 03:02:23.510178 9794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 03:02:23.521639 9799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:23.521569 9800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 03:02:23.521605 9802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 03:02:23.522460 9794 server_base.cc:1034] running on GCE node
I20260925 03:02:23.523236 9794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 03:02:23.524941 9794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 03:02:23.526163 9794 hybrid_clock.cc:648] HybridClock initialized: now 1790305343526112 us; error 27 us; skew 500 ppm
I20260925 03:02:23.526494 9794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 03:02:23 dist-test-slave-8zvk krb5kdc[9633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305343, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 03:02:23.529495 9794 init.cc:377] Logged in from keytab as kudu/127.18.196.252@KRBTEST.COM (short username kudu)
I20260925 03:02:23.530896 9794 webserver.cc:460] Webserver started at http://127.18.196.252:35945/ using document root <none> and password file <none>
I20260925 03:02:23.531606 9794 fs_manager.cc:359] Metadata directory not provided
I20260925 03:02:23.531713 9794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 03:02:23.532028 9794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 03:02:23.534746 9794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-2/data/instance:
uuid: "1e7ee4fa805640fb999cd02a8b25c509"
format_stamp: "Formatted at 2026-09-25 03:02:23 on dist-test-slave-8zvk"
server_key: "2e411b6048e0e34ccd785d0a0254a477"
server_key_iv: "5da61664a0478c3206ab22f450190d28"
server_key_version: "encryptionkey@0"
I20260925 03:02:23.535579 9794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-2/wal/instance:
uuid: "1e7ee4fa805640fb999cd02a8b25c509"
format_stamp: "Formatted at 2026-09-25 03:02:23 on dist-test-slave-8zvk"
server_key: "2e411b6048e0e34ccd785d0a0254a477"
server_key_iv: "5da61664a0478c3206ab22f450190d28"
server_key_version: "encryptionkey@0"
I20260925 03:02:23.540300 9794 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 03:02:23.543519 9809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 03:02:23.544685 9794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260925 03:02:23.544869 9794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-2/data,/tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-2/wal
uuid: "1e7ee4fa805640fb999cd02a8b25c509"
format_stamp: "Formatted at 2026-09-25 03:02:23 on dist-test-slave-8zvk"
server_key: "2e411b6048e0e34ccd785d0a0254a477"
server_key_iv: "5da61664a0478c3206ab22f450190d28"
server_key_version: "encryptionkey@0"
I20260925 03:02:23.545096 9794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 03:02:23.546425 9713 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } attempt: 3
I20260925 03:02:23.547785 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.547131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:41927 (local address 127.0.0.1:48700)
0925 03:02:23.547447 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.547548 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260925 03:02:23.549038 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.548606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:41927 (local address 127.0.0.1:48702)
0925 03:02:23.548796 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.548909 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260925 03:02:23.549304 9713 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.196.252:41927: Network error: Client connection negotiation failed: client connection to 127.18.196.252:41927: connect: Connection refused (error 111)
I20260925 03:02:23.568336 9794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 03:02:23.574050 9794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 03:02:23.574342 9794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 03:02:23.595278 9794 rpc_server.cc:307] RPC server started. Bound to: 127.18.196.252:41927
I20260925 03:02:23.595326 9860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.196.252:41927 every 8 connection(s)
I20260925 03:02:23.596776 9794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-2/data/info.pb
I20260925 03:02:23.601504 9861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 03:02:23.601552 19219 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu as pid 9794
I20260925 03:02:23.601727 19219 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM3GXH0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304987644417-19219-0/minicluster-data/master-2/wal/instance
I20260925 03:02:23.602298 19219 external_mini_cluster.cc:1428] Setting key 046b314a62cac966e7527720287e8e5d
I20260925 03:02:23.614332 9861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43313 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 03:02:23 dist-test-slave-8zvk krb5kdc[9633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.252@KRBTEST.COM
Sep 25 03:02:23 dist-test-slave-8zvk krb5kdc[9633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305343, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.252@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
I20260925 03:02:23.627517 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.604379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48708 (local address 127.18.196.252:41927)
0925 03:02:23.605239 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:23.605253 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:23.605300 (+ 47us) server_negotiation.cc:407] Connection header received
0925 03:02:23.607792 (+ 2492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:23.607820 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:23.608321 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:23.610368 (+ 2047us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:23.610867 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.612791 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.614117 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.614750 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.614999 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:23.619971 (+ 4972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:23.619980 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:23.619998 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:23.620054 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:23.623497 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.624201 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.624209 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.624217 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.624373 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.624763 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.624768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.624771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.625285 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:23.625659 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:23.626128 (+ 469us) server_negotiation.cc:299] Negotiation successful
0925 03:02:23.626498 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":395,"threads_started":1}
I20260925 03:02:23.634019 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.618761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56202 (local address 127.18.196.254:43313)
0925 03:02:23.618997 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:23.619004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:23.619727 (+ 723us) server_negotiation.cc:407] Connection header received
0925 03:02:23.619926 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:23.619933 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:23.620021 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:23.620132 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:23.621182 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.622343 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.623838 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.624262 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.624388 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:23.627592 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:23.627598 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:23.627604 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:23.627662 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:23.631051 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.631884 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.631890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.631893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.632053 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.632441 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.632465 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.632470 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.632823 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:23.633028 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:23.633480 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 03:02:23.633681 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 03:02:23.634037 9865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.618764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.254:43313 (local address 127.0.0.1:56202)
0925 03:02:23.619585 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.619624 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 03:02:23.619762 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:02:23.620165 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:02:23.620170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:02:23.620756 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:02:23.621028 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:23.621038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.622512 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.622516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:23.623688 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:23.623701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.624476 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.624483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:23.624647 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:23.625282 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:02:23.625304 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:02:23.627402 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:02:23.631161 (+ 3759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:23.631170 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:23.631174 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:23.631729 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:23.632192 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:23.632196 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:23.632198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:23.632314 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:23.633000 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:02:23.633007 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:02:23.633180 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 03:02:23.633346 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 03:02:23.633624 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":441,"threads_started":1}
I20260925 03:02:23.635502 9861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 03:02:23 dist-test-slave-8zvk krb5kdc[9633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.254@KRBTEST.COM
Sep 25 03:02:23 dist-test-slave-8zvk krb5kdc[9633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305343, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.252@KRBTEST.COM for kudu/127.18.196.253@KRBTEST.COM
I20260925 03:02:23.645059 9866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.630373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56214 (local address 127.18.196.254:43313)
0925 03:02:23.631011 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:23.631017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:23.631039 (+ 22us) server_negotiation.cc:407] Connection header received
0925 03:02:23.631082 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:23.631089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:23.631161 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:23.631249 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:23.632700 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.633511 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.635089 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.635653 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.635782 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:23.639737 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:23.639741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:23.639744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:23.639771 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:23.642249 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.642966 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.642972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.642975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.643154 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.643528 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.643534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.643537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.643872 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:23.644139 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:23.644404 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 03:02:23.644603 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":231,"threads_started":1}
I20260925 03:02:23.649326 9865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.636288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.253:41465 (local address 127.0.0.1:53664)
0925 03:02:23.636548 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.636561 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 03:02:23.636669 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:02:23.636978 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:02:23.636982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:02:23.637314 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:02:23.637442 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:23.637450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.638918 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.638930 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:23.639797 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:23.639808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.640531 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.640536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:23.640733 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:23.641176 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:02:23.641190 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:02:23.642975 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:02:23.646988 (+ 4013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:23.646991 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:23.646994 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:23.647366 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:23.647800 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:23.647804 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:23.647808 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:23.647942 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:23.648463 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:02:23.648468 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:02:23.648616 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 03:02:23.648743 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 03:02:23.648983 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260925 03:02:23.649482 9791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.636258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53664 (local address 127.18.196.253:41465)
0925 03:02:23.636476 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:23.636482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:23.636626 (+ 144us) server_negotiation.cc:407] Connection header received
0925 03:02:23.636787 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:23.636792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:23.636861 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:23.636961 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:23.637577 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.638206 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.639943 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.640381 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.640503 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:23.643161 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:23.643167 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:23.643171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:23.643227 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:23.646453 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.647493 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.647498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.647501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.647688 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.648073 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.648078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.648081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.648341 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:23.648495 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:23.648851 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 03:02:23.649072 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 03:02:23.650358 9861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 03:02:23 dist-test-slave-8zvk krb5kdc[9633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.196.253@KRBTEST.COM
Sep 25 03:02:23 dist-test-slave-8zvk krb5kdc[9633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305343, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.252@KRBTEST.COM for kudu/127.18.196.252@KRBTEST.COM
I20260925 03:02:23.662200 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.646650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53678 (local address 127.18.196.253:41465)
0925 03:02:23.647153 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:23.647157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:23.647186 (+ 29us) server_negotiation.cc:407] Connection header received
0925 03:02:23.647218 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:23.647221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:23.647262 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:23.647362 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:23.648602 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.649567 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.651075 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.651719 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.651843 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:23.656335 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:23.656340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:23.656343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:23.656369 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:23.659655 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.660321 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.660330 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.660335 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.660495 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.660857 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.660863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.660867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.661184 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:23.661445 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:23.661596 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 03:02:23.661787 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":221,"threads_started":1}
I20260925 03:02:23.664151 9865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.651119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:41927 (local address 127.0.0.1:48718)
0925 03:02:23.651260 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.651276 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 03:02:23.651366 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:02:23.651653 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:02:23.651658 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:02:23.652078 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:02:23.652256 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:23.652271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.653612 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.653616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:23.654419 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:23.654430 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.655222 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.655229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:23.655458 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:23.655956 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:02:23.655998 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:02:23.658197 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:02:23.661698 (+ 3501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:23.661703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:23.661707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:23.662133 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:23.662549 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:23.662553 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:23.662557 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:23.662709 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:23.663295 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:02:23.663301 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:02:23.663442 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 03:02:23.663540 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 03:02:23.663814 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 03:02:23.664386 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.651280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48718 (local address 127.18.196.252:41927)
0925 03:02:23.651427 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:23.651431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:23.651444 (+ 13us) server_negotiation.cc:407] Connection header received
0925 03:02:23.651478 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:23.651481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:23.651526 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:23.651593 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:23.652373 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.653476 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.654556 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.655056 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.655149 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:23.658359 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:23.658365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:23.658369 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:23.658402 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:23.661563 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.662268 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.662273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.662278 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.662410 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.662823 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.662827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.662829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.663180 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:23.663328 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:23.663647 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 03:02:23.664034 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":47}
I20260925 03:02:23.667022 19219 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 03:02:23.670199 9861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e7ee4fa805640fb999cd02a8b25c509: Bootstrap starting.
I20260925 03:02:23.670083 9688 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ca53c07fba764f5395d6d814c7e72ae5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56214
I20260925 03:02:23.673146 9861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e7ee4fa805640fb999cd02a8b25c509: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:23.674057 9861 log.cc:824] T 00000000000000000000000000000000 P 1e7ee4fa805640fb999cd02a8b25c509: Log is configured to *not* fsync() on all Append() calls
I20260925 03:02:23.676440 9861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e7ee4fa805640fb999cd02a8b25c509: No bootstrap required, opened a new log
I20260925 03:02:23.683117 9861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e7ee4fa805640fb999cd02a8b25c509 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca53c07fba764f5395d6d814c7e72ae5" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43313 } } peers { permanent_uuid: "9a8ee6fdb2e946f096ed94d56a4fea62" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } } peers { permanent_uuid: "1e7ee4fa805640fb999cd02a8b25c509" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } }
I20260925 03:02:23.683425 9861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e7ee4fa805640fb999cd02a8b25c509 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e7ee4fa805640fb999cd02a8b25c509, State: Initialized, Role: FOLLOWER
I20260925 03:02:23.683808 9861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e7ee4fa805640fb999cd02a8b25c509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca53c07fba764f5395d6d814c7e72ae5" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43313 } } peers { permanent_uuid: "9a8ee6fdb2e946f096ed94d56a4fea62" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } } peers { permanent_uuid: "1e7ee4fa805640fb999cd02a8b25c509" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } }
I20260925 03:02:23.684691 9869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e7ee4fa805640fb999cd02a8b25c509 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca53c07fba764f5395d6d814c7e72ae5" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43313 } } peers { permanent_uuid: "9a8ee6fdb2e946f096ed94d56a4fea62" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } } peers { permanent_uuid: "1e7ee4fa805640fb999cd02a8b25c509" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } } }
I20260925 03:02:23.685075 9869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e7ee4fa805640fb999cd02a8b25c509 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 03:02:23.685458 9861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e7ee4fa805640fb999cd02a8b25c509 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 03:02:23.694242 9880 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e7ee4fa805640fb999cd02a8b25c509: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 03:02:23.694370 9880 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 03:02:23.691792 9794 master_runner.cc:186] Error getting master registration for 127.18.196.254:43313: OK, instance_id { permanent_uuid: "ca53c07fba764f5395d6d814c7e72ae5" instance_seqno: 1790305342587767 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 03:02:23.702854 9794 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 03:02:23.726558 9713 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } attempt: 4
Sep 25 03:02:23 dist-test-slave-8zvk krb5kdc[9633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305342, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.254@KRBTEST.COM for kudu/127.18.196.252@KRBTEST.COM
I20260925 03:02:23.739737 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.727259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:41927 (local address 127.0.0.1:48730)
0925 03:02:23.727457 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.727477 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 03:02:23.727589 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:02:23.727945 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:02:23.727950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:02:23.728375 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:02:23.728566 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:23.728575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.729649 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.729655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:23.730620 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:23.730631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.731454 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.731460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:23.731658 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:23.732261 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:02:23.732278 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:02:23.734307 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:02:23.737275 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:23.737280 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:23.737285 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:23.737647 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:23.738023 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:23.738027 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:23.738031 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:23.738179 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:23.738714 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:02:23.738721 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:02:23.738935 (+ 214us) client_negotiation.cc:769] Sending connection context
0925 03:02:23.739082 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 03:02:23.739361 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 03:02:23.739746 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.727356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48730 (local address 127.18.196.252:41927)
0925 03:02:23.727562 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:23.727568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:23.727593 (+ 25us) server_negotiation.cc:407] Connection header received
0925 03:02:23.727719 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:23.727725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:23.727803 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:23.727919 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:23.728707 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.729500 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.730759 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.731291 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.731422 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:23.734481 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:23.734488 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:23.734492 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:23.734532 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:23.737138 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.737771 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.737777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.737781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.737909 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.738284 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.738290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.738293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.738596 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:23.738767 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:23.739203 (+ 436us) server_negotiation.cc:299] Negotiation successful
0925 03:02:23.739391 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 03:02:23.744027 9713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5: Bootstrap starting.
I20260925 03:02:23.748066 9713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:23.749315 9713 log.cc:824] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5: Log is configured to *not* fsync() on all Append() calls
I20260925 03:02:23.757618 9713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5: No bootstrap required, opened a new log
I20260925 03:02:23.765029 9713 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca53c07fba764f5395d6d814c7e72ae5" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43313 } } peers { permanent_uuid: "9a8ee6fdb2e946f096ed94d56a4fea62" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } } peers { permanent_uuid: "1e7ee4fa805640fb999cd02a8b25c509" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } }
I20260925 03:02:23.765422 9713 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca53c07fba764f5395d6d814c7e72ae5, State: Initialized, Role: FOLLOWER
I20260925 03:02:23.765884 9713 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca53c07fba764f5395d6d814c7e72ae5" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43313 } } peers { permanent_uuid: "9a8ee6fdb2e946f096ed94d56a4fea62" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } } peers { permanent_uuid: "1e7ee4fa805640fb999cd02a8b25c509" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } }
I20260925 03:02:23.766776 9883 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca53c07fba764f5395d6d814c7e72ae5" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43313 } } peers { permanent_uuid: "9a8ee6fdb2e946f096ed94d56a4fea62" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } } peers { permanent_uuid: "1e7ee4fa805640fb999cd02a8b25c509" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } } }
I20260925 03:02:23.767230 9883 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 03:02:23.767917 9713 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:02:23.768028 9688 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 03:02:23.768209 9688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:23.772112 9688 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca53c07fba764f5395d6d814c7e72ae5" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43313 } } peers { permanent_uuid: "9a8ee6fdb2e946f096ed94d56a4fea62" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } } peers { permanent_uuid: "1e7ee4fa805640fb999cd02a8b25c509" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } }
I20260925 03:02:23.775655 9836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca53c07fba764f5395d6d814c7e72ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1e7ee4fa805640fb999cd02a8b25c509"
I20260925 03:02:23.776160 9836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e7ee4fa805640fb999cd02a8b25c509 [term 0 FOLLOWER]: Advancing to term 1
W20260925 03:02:23.778703 9894 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 03:02:23.778908 9894 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 03:02:23.779314 9688 leader_election.cc:288] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5 [CANDIDATE]: Term 1 election: Requested vote from peers 9a8ee6fdb2e946f096ed94d56a4fea62 (127.18.196.253:41465), 1e7ee4fa805640fb999cd02a8b25c509 (127.18.196.252:41927)
I20260925 03:02:23.784062 9836 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1e7ee4fa805640fb999cd02a8b25c509 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca53c07fba764f5395d6d814c7e72ae5 in term 1.
I20260925 03:02:23.785390 9663 leader_election.cc:302] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e7ee4fa805640fb999cd02a8b25c509, ca53c07fba764f5395d6d814c7e72ae5; no voters:
I20260925 03:02:23.785804 9883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 03:02:23.780555 9646 master_runner.cc:186] Error getting master registration for 127.18.196.253:41465: OK, instance_id { permanent_uuid: "9a8ee6fdb2e946f096ed94d56a4fea62" instance_seqno: 1790305343078353 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 03:02:23.794428 9646 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 03:02:23.794577 9883 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5 [term 1 LEADER]: Becoming Leader. State: Replica: ca53c07fba764f5395d6d814c7e72ae5, State: Running, Role: LEADER
I20260925 03:02:23.795145 9883 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca53c07fba764f5395d6d814c7e72ae5" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43313 } } peers { permanent_uuid: "9a8ee6fdb2e946f096ed94d56a4fea62" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } } peers { permanent_uuid: "1e7ee4fa805640fb999cd02a8b25c509" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } }
I20260925 03:02:23.804278 9787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } attempt: 5
I20260925 03:02:23.812844 9897 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca53c07fba764f5395d6d814c7e72ae5. Latest consensus state: current_term: 1 leader_uuid: "ca53c07fba764f5395d6d814c7e72ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca53c07fba764f5395d6d814c7e72ae5" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43313 } } peers { permanent_uuid: "9a8ee6fdb2e946f096ed94d56a4fea62" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } } peers { permanent_uuid: "1e7ee4fa805640fb999cd02a8b25c509" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } } }
I20260925 03:02:23.813266 9897 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5 [sys.catalog]: This master's current role is: LEADER
Sep 25 03:02:23 dist-test-slave-8zvk krb5kdc[9633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305343, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.196.253@KRBTEST.COM for kudu/127.18.196.252@KRBTEST.COM
W20260925 03:02:23.827793 9667 tablet.cc:2369] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 03:02:23.827948 9815 tablet.cc:2369] T 00000000000000000000000000000000 P 1e7ee4fa805640fb999cd02a8b25c509: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 03:02:23.828210 9904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 03:02:23.835609 9790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.804887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.196.252:41927 (local address 127.0.0.1:48736)
0925 03:02:23.805172 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 03:02:23.805196 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 03:02:23.805328 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 03:02:23.805879 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 03:02:23.805884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 03:02:23.806407 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0925 03:02:23.806641 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:23.806654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.808170 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.808175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:23.809224 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 03:02:23.809236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.818991 (+ 9755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.818999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 03:02:23.819341 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:23.820123 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 03:02:23.820154 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 03:02:23.826010 (+ 5856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 03:02:23.832556 (+ 6546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:23.832561 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:23.832566 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:23.833102 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:23.833615 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 03:02:23.833619 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 03:02:23.833623 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 03:02:23.833789 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 03:02:23.834499 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 03:02:23.834505 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 03:02:23.834672 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 03:02:23.834828 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 03:02:23.835203 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260925 03:02:23.836627 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.804988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48736 (local address 127.18.196.252:41927)
0925 03:02:23.805163 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:23.805170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:23.805262 (+ 92us) server_negotiation.cc:407] Connection header received
0925 03:02:23.805483 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:23.805488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:23.805567 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:23.805666 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:23.806905 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.808034 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.809373 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.810049 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.810170 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:23.826659 (+ 16489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:23.826667 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:23.826672 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:23.826720 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:23.832417 (+ 5697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.833263 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.833269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.833272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.833471 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.833924 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.833929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.833931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.834278 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:23.834436 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:23.835993 (+ 1557us) server_negotiation.cc:299] Negotiation successful
0925 03:02:23.836238 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 03:02:23.837484 9904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 03:02:23.842564 9787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a8ee6fdb2e946f096ed94d56a4fea62: Bootstrap starting.
I20260925 03:02:23.848296 9787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a8ee6fdb2e946f096ed94d56a4fea62: Neither blocks nor log segments found. Creating new log.
I20260925 03:02:23.855321 9787 log.cc:824] T 00000000000000000000000000000000 P 9a8ee6fdb2e946f096ed94d56a4fea62: Log is configured to *not* fsync() on all Append() calls
I20260925 03:02:23.865536 9787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a8ee6fdb2e946f096ed94d56a4fea62: No bootstrap required, opened a new log
I20260925 03:02:23.879623 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.810826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48744 (local address 127.18.196.252:41927)
0925 03:02:23.846798 (+ 35972us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:23.846805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:23.846822 (+ 17us) server_negotiation.cc:407] Connection header received
0925 03:02:23.846887 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:23.846891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:23.846994 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:23.847095 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:23.849489 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.852086 (+ 2597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.854967 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.858066 (+ 3099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.867848 (+ 9782us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:23.868520 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:23.868527 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:23.868531 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:23.868594 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:23.874505 (+ 5911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.875457 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.875462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.875466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.875697 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.876224 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.876229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.876232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.876852 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:23.878213 (+ 1361us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:23.878434 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 03:02:23.878880 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35836,"thread_start_us":57190,"threads_started":1}
I20260925 03:02:23.882550 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.810487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53694 (local address 127.18.196.253:41465)
0925 03:02:23.810702 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:23.810708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:23.828077 (+ 17369us) server_negotiation.cc:407] Connection header received
0925 03:02:23.828725 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:23.828731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:23.828816 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:23.828887 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:23.837676 (+ 8789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.838753 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.845493 (+ 6740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.846170 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.846291 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:23.860045 (+ 13754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:23.860054 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:23.860061 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:23.860105 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:23.864168 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.879706 (+ 15538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.879712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.879715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.879864 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.880708 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.880715 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.880721 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.881063 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:23.881304 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:23.881981 (+ 677us) server_negotiation.cc:299] Negotiation successful
0925 03:02:23.882214 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 03:02:23.902027 9866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:23.818603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56224 (local address 127.18.196.254:43313)
0925 03:02:23.818821 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:23.818829 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:23.867692 (+ 48863us) server_negotiation.cc:407] Connection header received
0925 03:02:23.868414 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:23.868421 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:23.868510 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:23.868588 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:23.874953 (+ 6365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.876029 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.882433 (+ 6404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:23.883140 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:23.883243 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:23.893196 (+ 9953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 03:02:23.893202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 03:02:23.893209 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 03:02:23.893251 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 03:02:23.897112 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.899847 (+ 2735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.899853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.899857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.900003 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 03:02:23.900490 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 03:02:23.900494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 03:02:23.900497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 03:02:23.900857 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 03:02:23.901123 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:23.901459 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 03:02:23.901664 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 03:02:23.904575 9836 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e7ee4fa805640fb999cd02a8b25c509 [term 1 FOLLOWER]: Refusing update from remote peer ca53c07fba764f5395d6d814c7e72ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:02:23.905598 9883 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e7ee4fa805640fb999cd02a8b25c509" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:02:23.905646 9787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a8ee6fdb2e946f096ed94d56a4fea62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca53c07fba764f5395d6d814c7e72ae5" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43313 } } peers { permanent_uuid: "9a8ee6fdb2e946f096ed94d56a4fea62" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } } peers { permanent_uuid: "1e7ee4fa805640fb999cd02a8b25c509" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } }
I20260925 03:02:23.906327 9787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a8ee6fdb2e946f096ed94d56a4fea62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a8ee6fdb2e946f096ed94d56a4fea62, State: Initialized, Role: FOLLOWER
I20260925 03:02:23.906899 9787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a8ee6fdb2e946f096ed94d56a4fea62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca53c07fba764f5395d6d814c7e72ae5" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43313 } } peers { permanent_uuid: "9a8ee6fdb2e946f096ed94d56a4fea62" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } } peers { permanent_uuid: "1e7ee4fa805640fb999cd02a8b25c509" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } }
I20260925 03:02:23.917016 9869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e7ee4fa805640fb999cd02a8b25c509 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca53c07fba764f5395d6d814c7e72ae5. Latest consensus state: current_term: 1 leader_uuid: "ca53c07fba764f5395d6d814c7e72ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca53c07fba764f5395d6d814c7e72ae5" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43313 } } peers { permanent_uuid: "9a8ee6fdb2e946f096ed94d56a4fea62" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } } peers { permanent_uuid: "1e7ee4fa805640fb999cd02a8b25c509" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } } }
I20260925 03:02:23.917320 9869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e7ee4fa805640fb999cd02a8b25c509 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 03:02:23.928432 9910 mvcc.cc:205] Tried to move back new op lower bound from 7333090688620126208 to 7333090688190386176. Current Snapshot: MvccSnapshot[applied={T|T < 7333090688620126208}]
I20260925 03:02:23.930326 9787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a8ee6fdb2e946f096ed94d56a4fea62 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 03:02:23.930547 9883 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca53c07fba764f5395d6d814c7e72ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca53c07fba764f5395d6d814c7e72ae5" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43313 } } peers { permanent_uuid: "9a8ee6fdb2e946f096ed94d56a4fea62" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } } peers { permanent_uuid: "1e7ee4fa805640fb999cd02a8b25c509" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } } }
I20260925 03:02:23.932633 9883 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5 [sys.catalog]: This master's current role is: LEADER
I20260925 03:02:23.937216 9904 catalog_manager.cc:1347] Generated new cluster ID: ab732e46a4e24c4782a054278cd86651
I20260925 03:02:23.937319 9904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 03:02:23.937686 9869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e7ee4fa805640fb999cd02a8b25c509 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca53c07fba764f5395d6d814c7e72ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca53c07fba764f5395d6d814c7e72ae5" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43313 } } peers { permanent_uuid: "9a8ee6fdb2e946f096ed94d56a4fea62" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } } peers { permanent_uuid: "1e7ee4fa805640fb999cd02a8b25c509" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } } }
I20260925 03:02:23.937955 9869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e7ee4fa805640fb999cd02a8b25c509 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 03:02:23.930557 9762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca53c07fba764f5395d6d814c7e72ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9a8ee6fdb2e946f096ed94d56a4fea62"
I20260925 03:02:23.939805 9761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a8ee6fdb2e946f096ed94d56a4fea62 [term 0 FOLLOWER]: Advancing to term 1
I20260925 03:02:23.943394 9761 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9a8ee6fdb2e946f096ed94d56a4fea62 [term 1 FOLLOWER]: Refusing update from remote peer ca53c07fba764f5395d6d814c7e72ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 03:02:23.929412 9909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a8ee6fdb2e946f096ed94d56a4fea62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca53c07fba764f5395d6d814c7e72ae5" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43313 } } peers { permanent_uuid: "9a8ee6fdb2e946f096ed94d56a4fea62" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } } peers { permanent_uuid: "1e7ee4fa805640fb999cd02a8b25c509" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } } }
I20260925 03:02:23.944113 9909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a8ee6fdb2e946f096ed94d56a4fea62 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 03:02:23.948135 9883 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a8ee6fdb2e946f096ed94d56a4fea62" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 03:02:23.946768 9762 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9a8ee6fdb2e946f096ed94d56a4fea62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca53c07fba764f5395d6d814c7e72ae5 in term 1.
I20260925 03:02:23.958959 9904 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 03:02:23.960700 9904 catalog_manager.cc:1504] Loading token signing keys...
I20260925 03:02:23.984378 9904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5: Generated new TSK 0
I20260925 03:02:23.985423 9904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 03:02:24.016317 9920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a8ee6fdb2e946f096ed94d56a4fea62 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca53c07fba764f5395d6d814c7e72ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca53c07fba764f5395d6d814c7e72ae5" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43313 } } peers { permanent_uuid: "9a8ee6fdb2e946f096ed94d56a4fea62" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } } peers { permanent_uuid: "1e7ee4fa805640fb999cd02a8b25c509" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } } }
I20260925 03:02:24.016700 9920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a8ee6fdb2e946f096ed94d56a4fea62 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 03:02:24.017009 9883 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca53c07fba764f5395d6d814c7e72ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca53c07fba764f5395d6d814c7e72ae5" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43313 } } peers { permanent_uuid: "9a8ee6fdb2e946f096ed94d56a4fea62" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } } peers { permanent_uuid: "1e7ee4fa805640fb999cd02a8b25c509" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } } }
I20260925 03:02:24.017072 9909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a8ee6fdb2e946f096ed94d56a4fea62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca53c07fba764f5395d6d814c7e72ae5. Latest consensus state: current_term: 1 leader_uuid: "ca53c07fba764f5395d6d814c7e72ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca53c07fba764f5395d6d814c7e72ae5" member_type: VOTER last_known_addr { host: "127.18.196.254" port: 43313 } } peers { permanent_uuid: "9a8ee6fdb2e946f096ed94d56a4fea62" member_type: VOTER last_known_addr { host: "127.18.196.253" port: 41465 } } peers { permanent_uuid: "1e7ee4fa805640fb999cd02a8b25c509" member_type: VOTER last_known_addr { host: "127.18.196.252" port: 41927 } } }
I20260925 03:02:24.017307 9883 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca53c07fba764f5395d6d814c7e72ae5 [sys.catalog]: This master's current role is: LEADER
I20260925 03:02:24.017352 9909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a8ee6fdb2e946f096ed94d56a4fea62 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 03:02:24.065579 9931 catalog_manager.cc:1259] Loaded cluster ID: ab732e46a4e24c4782a054278cd86651
I20260925 03:02:24.065685 9931 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9a8ee6fdb2e946f096ed94d56a4fea62: loading cluster ID for follower catalog manager: success
I20260925 03:02:24.069175 9931 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9a8ee6fdb2e946f096ed94d56a4fea62: acquiring CA information for follower catalog manager: success
I20260925 03:02:24.071444 9931 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9a8ee6fdb2e946f096ed94d56a4fea62: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 03:02:24.121778 9866 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 03:02:24.110080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56238 (local address 127.18.196.254:43313)
0925 03:02:24.110296 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:24.110302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:24.119423 (+ 9121us) server_negotiation.cc:407] Connection header received
0925 03:02:24.119755 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:24.119761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:24.119862 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:24.119951 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:24.121386 (+ 1435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56238: BlockingRecv error: recv got EOF from 127.0.0.1:56238 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 03:02:24.124756 9867 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 03:02:24.111819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53710 (local address 127.18.196.253:41465)
0925 03:02:24.112081 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:24.112090 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:24.122748 (+ 10658us) server_negotiation.cc:407] Connection header received
0925 03:02:24.123085 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:24.123094 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:24.123204 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:24.123285 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:24.124437 (+ 1152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53710: BlockingRecv error: recv got EOF from 127.0.0.1:53710 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260925 03:02:24.127475 9864 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 03:02:24.120172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48756 (local address 127.18.196.252:41927)
0925 03:02:24.120356 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:24.120364 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:24.125581 (+ 5217us) server_negotiation.cc:407] Connection header received
0925 03:02:24.125853 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:24.125862 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:24.125960 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:24.126058 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:24.127183 (+ 1125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48756: BlockingRecv error: recv got EOF from 127.0.0.1:48756 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 03:02:24.176252 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:24.166895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53714 (local address 127.18.196.253:41465)
0925 03:02:24.167124 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:24.167134 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:24.167633 (+ 499us) server_negotiation.cc:407] Connection header received
0925 03:02:24.167986 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:24.167993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:24.168243 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:24.168380 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:24.169193 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.170357 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.173197 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.173892 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.173988 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:24.174725 (+ 737us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:24.175429 (+ 704us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:24.175571 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:24.175814 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 03:02:24.175904 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260925 03:02:24.223395 9864 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 03:02:24.209984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48758 (local address 127.18.196.252:41927)
0925 03:02:24.210201 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:24.210208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:24.221175 (+ 10967us) server_negotiation.cc:407] Connection header received
0925 03:02:24.221724 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:24.221731 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:24.221890 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:24.221975 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:24.223131 (+ 1156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48758: BlockingRecv error: recv got EOF from 127.0.0.1:48758 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 03:02:24.277309 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:24.259084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53724 (local address 127.18.196.253:41465)
0925 03:02:24.259270 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:24.259276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:24.270018 (+ 10742us) server_negotiation.cc:407] Connection header received
0925 03:02:24.270536 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:24.270542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:24.270637 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:24.270703 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:24.271533 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.272724 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.274931 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.275670 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.275809 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:24.276429 (+ 620us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:24.276599 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:24.276677 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:24.276903 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 03:02:24.277008 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 03:02:24.323264 9864 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 03:02:24.310488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48760 (local address 127.18.196.252:41927)
0925 03:02:24.310708 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:24.310715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:24.321278 (+ 10563us) server_negotiation.cc:407] Connection header received
0925 03:02:24.321746 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:24.321752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:24.321865 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:24.321960 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:24.323002 (+ 1042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48760: BlockingRecv error: recv got EOF from 127.0.0.1:48760 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 03:02:24.348459 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:24.341052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53728 (local address 127.18.196.253:41465)
0925 03:02:24.341246 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:24.341253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:24.341833 (+ 580us) server_negotiation.cc:407] Connection header received
0925 03:02:24.342070 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:24.342076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:24.342177 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:24.342286 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:24.342804 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.343993 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.346102 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.346853 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.346997 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:24.347514 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:24.347691 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:24.347810 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:24.348091 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 03:02:24.348174 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 03:02:24.357155 9864 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 03:02:24.354925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48774 (local address 127.18.196.252:41927)
0925 03:02:24.355107 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:24.355112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:24.355630 (+ 518us) server_negotiation.cc:407] Connection header received
0925 03:02:24.355805 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:24.355811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:24.355936 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:24.356050 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:24.356919 (+ 869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48774: BlockingRecv error: recv got EOF from 127.0.0.1:48774 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 03:02:24.375718 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:24.368672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53744 (local address 127.18.196.253:41465)
0925 03:02:24.368885 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:24.368891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:24.369433 (+ 542us) server_negotiation.cc:407] Connection header received
0925 03:02:24.369613 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:24.369619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:24.369708 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:24.369809 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:24.370231 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.371450 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.373588 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.374292 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.374435 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:24.374921 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:24.375116 (+ 195us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:24.375209 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:24.375369 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 03:02:24.375452 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260925 03:02:24.386993 9864 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 03:02:24.384500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48776 (local address 127.18.196.252:41927)
0925 03:02:24.384756 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:24.384762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:24.385262 (+ 500us) server_negotiation.cc:407] Connection header received
0925 03:02:24.385438 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:24.385444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:24.385542 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:24.385644 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:24.386727 (+ 1083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48776: BlockingRecv error: recv got EOF from 127.0.0.1:48776 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
I20260925 03:02:24.405767 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:24.398794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53746 (local address 127.18.196.253:41465)
0925 03:02:24.398998 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:24.399005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:24.399579 (+ 574us) server_negotiation.cc:407] Connection header received
0925 03:02:24.399765 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:24.399771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:24.399897 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:24.399998 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:24.400507 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.401673 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.403660 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.404344 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.404469 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:24.404946 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:24.405076 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:24.405162 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:24.405329 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 03:02:24.405425 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260925 03:02:24.414494 9864 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 03:02:24.412721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48786 (local address 127.18.196.252:41927)
0925 03:02:24.412895 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:24.412901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:24.412919 (+ 18us) server_negotiation.cc:407] Connection header received
0925 03:02:24.413112 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:24.413118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:24.413205 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:24.413271 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:24.414240 (+ 969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48786: BlockingRecv error: recv got EOF from 127.0.0.1:48786 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 03:02:24.444199 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:24.434354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.18.196.253:41465)
0925 03:02:24.434563 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:24.434570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:24.438992 (+ 4422us) server_negotiation.cc:407] Connection header received
0925 03:02:24.439167 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:24.439172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:24.439264 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:24.439374 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:24.439787 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.440925 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.442234 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.442925 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.443049 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:24.443487 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:24.443608 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:24.443699 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:24.443820 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 03:02:24.443897 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 03:02:24.453889 9864 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 03:02:24.451876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48790 (local address 127.18.196.252:41927)
0925 03:02:24.452111 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:24.452116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:24.452461 (+ 345us) server_negotiation.cc:407] Connection header received
0925 03:02:24.452637 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:24.452641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:24.452709 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:24.452790 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:24.453627 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48790: BlockingRecv error: recv got EOF from 127.0.0.1:48790 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 03:02:24.495633 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:24.489504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53758 (local address 127.18.196.253:41465)
0925 03:02:24.489731 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:24.489740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:24.490135 (+ 395us) server_negotiation.cc:407] Connection header received
0925 03:02:24.490296 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:24.490302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:24.490409 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:24.490553 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:24.490917 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.492185 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.493577 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.494258 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.494400 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:24.494784 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:24.494984 (+ 200us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:24.495081 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:24.495223 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 03:02:24.495306 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260925 03:02:24.505373 9864 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 03:02:24.503224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.18.196.252:41927)
0925 03:02:24.503426 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:24.503434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:24.503964 (+ 530us) server_negotiation.cc:407] Connection header received
0925 03:02:24.504147 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:24.504163 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:24.504243 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:24.504319 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:24.505167 (+ 848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48804: BlockingRecv error: recv got EOF from 127.0.0.1:48804 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 03:02:24.578935 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:24.573104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53760 (local address 127.18.196.253:41465)
0925 03:02:24.573373 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:24.573382 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:24.573798 (+ 416us) server_negotiation.cc:407] Connection header received
0925 03:02:24.573970 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:24.573976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:24.574062 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:24.574167 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:24.574546 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.575703 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.576991 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.577654 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.577774 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:24.578129 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:24.578270 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:24.578353 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:24.578507 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 03:02:24.578587 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260925 03:02:24.587131 9864 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 03:02:24.585075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48808 (local address 127.18.196.252:41927)
0925 03:02:24.585233 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:24.585237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:24.585668 (+ 431us) server_negotiation.cc:407] Connection header received
0925 03:02:24.585899 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:24.585905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:24.585991 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:24.586097 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 03:02:24.586898 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48808: BlockingRecv error: recv got EOF from 127.0.0.1:48808 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 03:02:24.696336 9880 catalog_manager.cc:1259] Loaded cluster ID: ab732e46a4e24c4782a054278cd86651
I20260925 03:02:24.696439 9880 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1e7ee4fa805640fb999cd02a8b25c509: loading cluster ID for follower catalog manager: success
I20260925 03:02:24.698967 9880 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1e7ee4fa805640fb999cd02a8b25c509: acquiring CA information for follower catalog manager: success
I20260925 03:02:24.700876 9880 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1e7ee4fa805640fb999cd02a8b25c509: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 03:02:24.723152 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:24.718336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53776 (local address 127.18.196.253:41465)
0925 03:02:24.718588 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:24.718594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:24.718811 (+ 217us) server_negotiation.cc:407] Connection header received
0925 03:02:24.719065 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:24.719070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:24.719135 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:24.719234 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:24.719615 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.720305 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.721532 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.721994 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.722104 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:24.722435 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:24.722539 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:24.722608 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:24.722732 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 03:02:24.722794 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260925 03:02:24.734107 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:24.728674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48818 (local address 127.18.196.252:41927)
0925 03:02:24.728854 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:24.728863 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:24.729183 (+ 320us) server_negotiation.cc:407] Connection header received
0925 03:02:24.729383 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:24.729387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:24.729521 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:24.729641 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:24.730004 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.730712 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.732001 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.732474 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.732607 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:24.733064 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:24.733562 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:24.733711 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:24.733769 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 03:02:24.733847 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 03:02:24.747948 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:24.739998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53780 (local address 127.18.196.253:41465)
0925 03:02:24.740221 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:24.740229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:24.741331 (+ 1102us) server_negotiation.cc:407] Connection header received
0925 03:02:24.741593 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:24.741601 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:24.741704 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:24.741793 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:24.742571 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.743550 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.745738 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.746270 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.746351 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:24.747105 (+ 754us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:24.747250 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:24.747315 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:24.747542 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 03:02:24.747623 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 03:02:24.747964 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:24.739709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48820 (local address 127.18.196.252:41927)
0925 03:02:24.740438 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:24.740446 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:24.741137 (+ 691us) server_negotiation.cc:407] Connection header received
0925 03:02:24.741435 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:24.741440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:24.741527 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:24.741591 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:24.742527 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.743387 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.745751 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.746461 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.746594 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:24.747167 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:24.747295 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:24.747354 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:24.747649 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 03:02:24.747726 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546}
I20260925 03:02:24.750092 10035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 03:02:24.739720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56246 (local address 127.18.196.254:43313)
0925 03:02:24.740240 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0925 03:02:24.740249 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 03:02:24.741698 (+ 1449us) server_negotiation.cc:407] Connection header received
0925 03:02:24.741964 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 03:02:24.741975 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 03:02:24.742134 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 03:02:24.742241 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 03:02:24.742980 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.743958 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.747068 (+ 3110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 03:02:24.747646 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 03:02:24.747762 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 03:02:24.748968 (+ 1206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 03:02:24.749558 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 03:02:24.749642 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 03:02:24.749760 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 03:02:24.749823 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":733,"threads_started":1}
I20260925 03:02:24.754114 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 9646
I20260925 03:02:24.755198 9708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:02:24.871992 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 9646
I20260925 03:02:24.893390 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 9720
I20260925 03:02:24.894539 9782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:02:25.000943 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 9720
I20260925 03:02:25.021461 19219 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu pid 9794
I20260925 03:02:25.022610 9856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 03:02:25.126703 19219 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM3GXH0/build/asan/bin/kudu with pid 9794
2026-09-25T03:02:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4308 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8597 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (357417 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST